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Abstract 


The National Math + Science Initiative’s (NMSI’s) College Readiness Program (CRP) 
is an established program whose goal is to promote science, technology, engineering, 
and mathematics education in high schools to improve students’ readiness for college. 
The program provides teacher, student, and school supports to promote high school 
students’ success in mathematics, science, and English Advanced Placement (AP) 
courses, with a focus on students who are traditionally underrepresented in the targeted 
AP courses. 


Through a scale-up grant awarded to NMSI by the Investing in Innovation (3) 
program, the CRP was implemented in 28 schools in the 2016-2017 school year. 
CRESST conducted an independent evaluation of the impact of the CRP on students’ 
AP outcomes using a randomized cluster trial with 28 CRP schools and 24 control 
schools in 10 states. The evaluation of the CRP consisted of two parts: (1) assessment 
of the program’s impact on selected student AP exam outcomes and (2) assessment of 
the fidelity of implementation of the CRP. 


Program impact was evaluated using a 2-level hierarchical generalized linear model 
(HGLM) with students nested within schools The descriptive statistics showed that a 
higher percetange of students in the treatment schools took at least one AP course 
(30.7%) compared to those in the control schools (26.4%) by approximately 4.3%, 
however the difference was not statistically significant. In addition, students in the 
treatment schools were not more likely to achieve a score of 3 or higher, when 
compared to the delayed treatment schools. We further examined the effectiveness of 
the CRP using the prior year’s school-level performance on the AP exam as a covariate. 
As with the above findings, the results indicated the probability of a student taking at 
least one AP course or scoring 3 or higher on at least one AP exam is not statistically 
different between students in the treatment schools and those in the control treatment 
schools. 


Fidelity of implementation was evaluated using a fidelity matrix approach (required as 
part of the evaluation of the 13 program), which showed that not all elements of the 
program were implemented with high fidelity. Overall results, however, indicated that 
23 schools out of 28 treatment schools (82.1%) achieved 80% or better implementation 
fidelity, for an average fidelity score of 89.5%. Seven schools achieved a perfect 100% 
fidelity score. Looking at the different indicator groups (school, teacher and student), 


we found that all school support measures across all schools were implemented with 
fidelity. In over 80% of schools, not all teachers fulfilled their requirements for 
attending all training sessions, and so this component was not implemented with 
fidelity. Stipends and teacher awards were paid as expected as were student award 
payments. 


Teacher survey data indicated that teachers found the training and professional 
development activities provided by the CRP to be the most benefical program supports 
relating to helping increase student achievement in AP courses. Teacher incentives 
were chosen as the least important program component relating to increasing student 
performance by 16% of teachers and student incentives by 12% of teachers. Teachers 
did, however, view the student incentives as an important program component to 
encourage enrollment in AP courses. Likewise, students rated the financial incentives 
on average as somewhat important in encouraging them to participate in AP courses. 


I. Introduction 


Proficiency in math and science is crucial to our country’s capacity for innovation and 
future economic growth, yet a growing number of students lack foundational knowledge and 
skills in these subjects. Performance in math and science of U.S. college students is also below 
that of their peers in many other nations (Chen, 2013; Fleischman, Hopstock, Pelczar, & Shelley, 
2010). In 2011, for example, roughly one third of U.S. bachelor’s degrees were awarded in 
science and engineering fields, compared to 60% in Japan and 50% in China (National Science 
Board, 2014). Indeed, it is estimated that in 2016, only around 41% of U.S. high school 
graduates were ready for college-level math, and only 36% ready for college-level science (ACT, 
Inc., 2016). The most recent Programme for International Student Assessment (PISA) results, 
from 2015, found the U.S. placed 38th of 71 countries in math and 34th in science (Pew 
Research Center, 2017). The necessity for an increased focus on math and science specifically is 
based on years of research which shows fewer students are entering math- and science-related 


career fields (National Science Board, 2010). 


These issues are even more pronounced for high-need and traditionally underserved 
students who may face hurdles because of policies and mindsets that limit their ability to access 
rigorous coursework. Data from the National Science Foundation (NSF) found that 27% of ninth 
graders in the lowest socioeconomic status category were not enrolled in any science courses, 
compared with 11% of students in the highest income category. These differences in access and 
opportunity can lead to achievement gaps that continue through college and beyond. The gap 
between White students’ six-year college graduation rates and their African American peers is 
around 22%, and the gap between White students and their Hispanic peers is 10% (Kena et al., 
2014). 


The National Math and Science Initiative (NMSI) was formed to address the declining 
number of students prepared to take rigorous college courses in math and science and equipped 
for careers in those fields. The College Readiness Program (CRP) was created to raise the 
academic bar in public schools by demonstrating that more students, especially high-need 
students, can master rigorous Advanced Placement (AP®) coursework, with a particular 
emphasis on math and science. The CRP addresses the need to improve STEM education, 
increase academic intensity, and improve student achievement in order to decrease the college 


readiness gap, especially among traditionally underrepresented and high-need students. 


Program supports include teacher training; incentive payments to teachers, administrators, 
and students (tied to AP performance); student study sessions; and provision of equipment and 
supplies. Over a three-year period, the CRP supports high schools, focusing on school reform, 
changing school culture, building a pipeline of students who enter high school ready for more 
rigorous coursework, and working within the existing school framework (as opposed to creating 


new schools). 


This project builds on several prior evaluations, most recently the success of NMSI’s 2011 
validation i3 grant which focused on schools in Colorado and Indiana. This study aimed to 
explore the impact of NMSI’s CRP on selected student outcomes and evaluate the fidelity of 
implementation of the program in 10 states. This report presents findings from the evaluation of 
the impact of the CRP as well as findings from the fidelity of implementation study based on the 


fidelity matrix approach. 


In the next section we provide an overview of the CRP and key components which were 


the focus of the analysis of the fidelity of program implementation in this evaluation. 


II. Program Description 


A. CRP Logic Model 


The CRP logic model (Figure 1) presents the key components of the intervention: program 
management, teacher support, student support, and financial awards. For teachers, the program 
offers (a) course-specific training, (b) access to expert mentors, and (c) online curricular 
resources. For students, the program offers (a) study sessions focused on student support, (b) 
exam fee subsidies, and (c) access to classroom materials needed to support rigorous 
coursework. For schools, the program offers (a) an annual review of program components and 
compliance to ensure maximum program effectiveness, and support of academic and program 
experts; (b) mutually agreed upon expectations for program participation and support; and (c) 
performance goals for teachers, students, and schools. At all levels there are also financial 


incentives associated with participating and succeeding in AP courses/exams. 


Teacher participation in professional development and mentoring, their access to rigorous 
materials and resources, and the use of incentives are designed to drive increased knowledge and 
use of pedagogical strategies as well as increased content knowledge and increased effectiveness 
in the classroom. Those intermediate outcomes should drive longer term outcomes such as 
increases in AP course enrollment, increases in the percentage of qualifying AP scores, and an 


increased number of teachers qualified to teach AP courses. 


For students, additional time on task, access to rigorous materials, resources, homework 
help, awards for performance, and exposure to highly trained teachers are all designed to 
increase student engagement, preparation, and motivation to perform well. These intermediate 
outcomes should then influence AP enrollment and percentage of qualifying AP scores. 
Furthermore, more students at a school obtaining qualifying scores and having positive 
experiences in AP courses should positively impact the number of students persisting in STEM 


courses. 


Key Components Mediators Outcomes 


= 


Figure 1. Logic model supporting NMSI’s College Readiness Program. 


B. Program Structure 


The program begins with an intensive summer teacher training for AP teachers. Since 
teacher training has limited effectiveness without additional support, NMSI AP teachers are also 
given access to an expert mentor to provide coaching, support, and assistance throughout the 
year. Teachers must apply for a mentor and once they have done so are assigned an online 
mentor with whom they interact on a weekly basis. Some mentors visit teachers’ classrooms 
during the academic year to provide opportunities for CRP teachers to observe a more 
experienced “master teacher.” CRP schools also have use of in-depth, online content for teachers 


and students to maximize their success. 


The next component of the program provides more time on task for students. Students 
attend three 6-hour Saturday study sessions taught by a master AP teacher—time that equates to 
three extra weeks of AP class time. The study sessions provide professional development and 
collaboration opportunities; teachers and students from multiple schools participate and teachers 
can see how expert teachers address difficult parts of AP courses. For the three years of NMSI 
program implementation, teachers continue to receive progressively more rigorous training and 
lessons; teachers and administrators continue to push further toward increasingly challenging 
goals; and both students and teachers receive nominal monetary awards for success. NMSI staff 
work with teachers and administrators throughout implementation to track progress toward their 


goals and troubleshoot where needed. 


III. Research Questions 


The evaluation of the CRP under the 13 scale-up grant consisted of two parts: (1) 
assessment of the program’s impact on selected student AP exam outcomes and (2) assessment 
of the fidelity of implementation of the CRP. The evaluation included 28 treatment and 24 


control schools in 10 states where CRP was being implemented as part of the 13 scale-up grant. 


Research questions for the evaluation were as follows: 


1. What was the impact of the CRP on the likelihood that students took STEM-related AP 
courses? 


2. What was the impact of the CRP on the likelihood that students achieved a qualifying 
score of 3 or higher on STEM-related AP exams? 


3. To what extent were each of the key components of the CRP implemented with 
fidelity? 


4. What were the facilitators and barriers to implementation? 


A. Study Design and Sample 


To address our research questions, we conducted a randomized cluster trial (RCT). Schools 
were randomly assigned either to treatment or control conditions and teachers within schools 
following the school assignment. Schools were randomly assigned to experimental conditions 
prior to program implementation. In the year on which this report is based (2016-2017), there 
were two groups of schools: an initial treatment group and a delayed treatment group (i.e., 
control schools in year 2016-2017). The delayed treatment group comprised schools in which 


implementation of CRP involved a one-year delay. 


For the RCT in year 2016-2017, comparisons were made between the treatment schools 
and the delayed treatment schools as control schools. For 2017—2018, both initial treatment 
schools and delayed treatment schools will constitute the treatment group and comparison 


schools will be selected using propensity score matching techniques. 


For the RCT, we first sampled 10 different regions (CA, GA, IL, LA, MI, MO, ND, OH, 
PA, and TX). These regions represented all locations where the CRP was being implemented in 
this scale-up grant. Within eight regions/states, one target school district was recruited, while two 
regions/states recruited more than one district: three districts for PA, and two for ND. The target 
districts included those with (1) a high concentration of socioeconomically disadvantaged 


students, and (2) high schools which have provided very few or no AP courses in the past. 


Within each region and district, schools were invited to apply for participation. Those 
schools that completed the application were randomized to treatment or delayed treatment 
conditions within district and region, as shown in Table 1. Fifty-two schools were randomly 
assigned to either treatment or delayed treatment groups in Year 1. In Year 1, delayed treatment 
schools conducted business-as-usual (BAU). That is, they utilized standard methods of providing 
AP courses, which typically do not include the supports NMSI provides to teachers and students, 


including incentives for successful completion of AP tests. 


Note randomization within district and region led to unequal numbers of schools in 
treatment and control conditions in some regions. In ND, we randomly assigned two schools to 
the treatment condition and one school to the delayed treatment condition in one district, and in 
the second district, one treatment and one delayed treatment school were assigned. In PA, there 
were originally 10 schools from 10 different school districts while we were conducting random 
assignment in March 2016. We divided 10 schools into two regions (eight for one region and two 
for the other region) in a geographically clustered fashion. Within each region, we performed 
random assignment, four schools for treatment and four schools for the delayed treatment 


condition in the first region and one for treatment and one for the delayed treatment condition in 


the second region. However, one treatment school in the first region and one treatment school in 
the second region did not participate in this study, and it resulted in three treatment schools and 
five delayed treatment schools participating in PA. Later, three delayed treatment schools 
withdrew, which resulted in three treatment schools and two delayed treatment schools. Also 
note that these regions represent blocks which were taken into account in our impact model. 


Table 1 below shows the numbers of regions, schools, and students in this study. 


Table | 
Sampling Plan for RCT 


School Students (11th and 12th graders) 


Region Treatment Delayed treatment Treatment Delayed treatment 


CA 2 1 1,097 689 
GA 2 2 826 1,483 
IL 3 2 829 587 
LA 1 1 523 263 
MI 2 1 1,618 959 
MO 3 3 791 376 
ND 3 2 1,648 1,260 
OH 4 4 888 644 
PA 3 2 1,593 890 
TX be) 3 2,553 1,770 
Total 28 21 12,366 8,921 


IV. Results for Impact Study 
A. Descriptive Statistics of 2017 AP Course Taking and Exam Data 


We first provide descriptive statistics of the AP course taking and exam data. These 
descriptive statistics present the percentage of each outcome of interest by treatment and delayed 
treatment schools (see Table 2 through Table 7). Note that the data on AP courses taken by 


students as well as AP exam data are based only on the enrolled 11" and 12" grade students. 


Students in the treatment schools were more likely to take an AP course than those in the 
delayed treatment schools. In the treatment schools 30.7% of the total enrolled 11 and 12" 
grade students took at least one AP course, compared to 26.4% of students in the delayed 
treatment schools (see Table 2). However, the percentage of students who scored 3 or higher on 


at least one AP exam was higher in the delayed treatment group (9.4% of enrolled students and 


35.7% of the students who took an AP course) than in the treatment group (7.3% and 23.6% 
respectively). This suggests that students in the treatment schools were slightly less likely to 


score 3 or higher on an AP exam than those in the delayed treatment schools. 


Table 2 


Outcomes for Students Taking (or Scoring 3+) on at Least One AP Course 


Students taking AP course Students with score of 3+ 


Group n % n % As % of course takers 
Treatment 3794 30.7% 897 7.3% 23.6% 
Delayed treatment 2353 26.4% 840 9.4% 35.7% 
All schools 6147 28.9% 1737 8.2% 28.3% 


Table 3 shows that the percentage of students who took at least one math AP course was 
slightly higher in the treatment schools (7.9%) than the delayed treatment schools (7.6%), 
whereas the percentage of students who scored 3 or higher on at least one math AP exam was 
higher in the delayed treatment schools (3.1% of all enrolled 11 and 12" grade students and 
41.4% of the students who took any AP math course) than the percentage of students in the 
treatment schools (2.3%; 29.4%). 


Table 3 
Outcome for Students Taking (or Scoring 3+) on at Least One Math AP Course 


Students taking AP course 


Students with score of 3+ 


Group n % n % As % of course takers 
Treatment OS) 7.9% 287 2.3% 29.4% 
Delayed treatment 674 7.6% 279 3.1% 41.4% 
All schools 1649 7.7% 566 2.7% 34.3% 


For science (see Table 4), the percentage of students who took at least one science AP 


course was higher in the treatment schools (13.7%) than the delayed treatment schools (11.2%). 


In contrast, the delayed treatment schools showed a higher percentage of students who scored 3 


or higher on at least one AP science exam (2.7% of all enrolled students and 23.9% of the 


students who took any AP science course) than the treatment schools (2.2%; 15.9%). 


Table 4 


Outcomes for Students Taking (or Scoring 3+) on at Least One AP Science Course 


Students taking AP course Students with score of 3+ 
Group n % n % As % of course takers 
Treatment 1697 13.7% 269 2.2% 15.9% 
Delayed treatment 996 11.2% 238 2.7% 23.9% 
All schools 2693 12.7% 507 2.4% 18.8% 


Similar results are shown in Table 5. The percentage of students who took at least one AP 
English course was higher in the treatment schools (18%) than the delayed treatment schools 
(13.6%). But the percentage of students who scored 3 or higher on at least one AP English exam 
was higher in the delayed treatment schools (5.1% of all enrolled students and 37.3% of the 


students who took any AP English course) than the percentage of students in the treatment 
schools (3.9%; 21.6%). 


Table 5 


Outcomes for Students Taking (or Scoring 3+) on at Least One English AP Course 


Students taking AP course Students with score of 3+ 
Group n % n % As % of course takers 
Treatment 2222 18.0% 481 3.9% 21.6% 
Delayed treatment 1212 13.6% 452 5.1% 37.3% 
All schools 3434 16.1% 933 44% 27.2% 


Likewise, when combining students who took at least one math or science AP course, the 
percentage of students taking at least one of these courses was higher in the treatment schools 
(17.7%) compared to the delayed treatment schools (15%). Conversely, the delayed treatment 
schools had a higher percentage of students who scored 3 or higher on at least one math or 
science AP exam (4.7% of all enrolled students and 31.6% of the students who took any math or 
science AP course) than the treatment schools (3.7%; 20.7%) (see Table 6). 


Table 6 
Outcome for Students Taking (or Scoring 3+) at Least One Math AP or Science AP Course 


Students taking AP course Students with score of 3+ 
Group n % n % As % of course takers 
Treatment 2195 17.7% 455 3.7% 20.7% 
Delayed treatment 1340 15.0% 423 4.7% 31.6% 
All schools 3535 16.6% 878 4.1% 24.8% 


For the math, science, or English subject areas combined (see Table 7), the percentage of 
students who took at least one math, science, or English AP course was higher in the treatment 
schools (27.2%) than the delayed treatment schools (22.1%). In contrast, the percentage of 
students who scored 3 or higher on at least one math, science, or English AP exam was higher in 
the delayed treatment schools (8% of all enrolled students and 36% of the students who took any 
AP course in this category) than the percentage of students in the treatment schools (6.2%; 
22.17). 


Table 7 


Outcomes for Students Taking (or Scoring 3+) at Least One Math, Science, or English AP Course 


Students taking AP course Students with score of 3+ 
Group n % n % As % of course takers 
Treatment 3368 27.2% 764 6.2% 22.7% 
Delayed treatment 1972 22.1% 710 8.0% 36.0% 
All schools 5340 25.1% 1474 6.9% 27.6% 


Figures 2 and 3 summarize the percentage of students who took at least one AP course in 
each subject category and the percentage of students who scored 3 or higher on at least one AP 
exam in each subject category, respectively. Figure 2 illustrates that a higher percentage of 
students in the treatment schools took at least one AP exam when compared with students in the 
delayed treatment schools across all subject area categories. The difference in percentage 
between the treatment and delayed treatment schools was largest for the math, science, or 


English category (5.1%) and smallest for math alone (0.3%). 


In contrast, Figure 3 reveals a different pattern for the percentages of students who scored 3 


or higher on at least one AP exam. Students in the treatment schools were slightly less likely to 
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score 3 or higher on at least one AP exam than students in the delayed treatment schools for 
every subject category. The differences in percentages of the outcomes among the students who 
enrolled in an AP course were very small between the treatment and delayed treatment schools, 
ranging from 0.5% for science to 2.1% when considering all subject areas together. As reported 
in Tables 1 to 6, however, when we look at student outcomes (score of 3 or higher) as a 
percentage of the number of students who took the course, there were sizable differences 
between the treatment and delayed treatment schools. In all subject areas, a greater percentage of 
course takers in the delayed treatment schools scored a 3 or higher on the AP exam, with the 


largest difference for English (15.7%) and the smallest for science (8%). 


35.0% 30.7% 


30.0% 6.4% 27.2% 
25.0% als 
17.7% 18.0% 
9, 

20.0% 15.0% 13.7% 3.6% 
15.0% aed 
10.0% i 7.9%7 6% 

5.0% i [| 
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Figure 2. Percentage of students who took at least one AP course in each subject category. 
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Figure 3. Percentage of students who scored 3 or higher on at least one AP exam in each subject category. 
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B. Statistical Model: 2-Level Hierarchical Generalized Linear Model 


We used a 2-level hierarchical generalized linear model (HGLM) with students nested 
within schools to estimate the effectiveness of the CRP on the likelihood of a student taking AP 
courses and attaining a qualifying score of 3 or higher on AP exams. The effectiveness of the 
CRP was examined in six different subject categories: (a) math, (b) science, (c) English, (d) math 


or science, (e) math, science, or English, and (f) any AP courses. 


The primary outcomes for the 2-level HGLM were binary: (1) whether or not a student 
took at least one AP course (coded as 0 for not taking an AP course; | for taking one or more AP 
courses), and (2) whether or not a student earned a qualifying score of 3 or higher on at least one 
AP exam in a subject category (coded as 0 for not achieving qualifying scores in AP exams; | for 


achieving qualifying scores in AP exams). 


The 2-level HGLM included one school-level treatment indicator variable. To increase the 
precision of the treatment effect estimate, this model was further extended by adding the prior 
year’s school-level performance on the AP exam as a covariate. Thus, for each outcome variable, 
two HGLMs were analyzed: (1) 2-level HGLM with the school-level treatment indicator; and (2) 
2-level HGLM with the school-level treatment indicator and the school-level performance on the 
2016 AP exam. Table 8 provides the complete list of the outcome variables for each subject 
category from the 2017 AP exam data and their respective covariates from the 2016 AP exam 
data. 


a. 2-Level HGLM With a School-Level Treatment Indicator 


A binary response Yix was the outcome for student i either in the 11" or 12" grade in school 
k. As the outcomes of interest were binary, we used a binomial sampling, defining the probability 
of the response equal to one as @ik= Pr (Yix =1) and let @ix be modeled using a logit link function. 
Note that we have two outcomes of interest: a) binary response of whether a student took an AP 
course; b) binary response of whether a student’s AP exam score were 3 or higher. Analyses 
were conducted separately for each outcome. The level 1 model was specified as follows: 


Pik 
= T0k, la 
ge (la) 


Nix = log ( 


At level 2, the school-level treatment indicator Trtx was included. Trtz was coded 0 for 
comparison schools (i.e., delayed treatment schools) and 1 for CRP schools (i.e., treatment 
schools). The Level 2 model was defined as: 

Tok = Yoo + yorTrtk + rok rox ~ N(0, vo) (1b) 
yoo represents the average log-odds of students taking at least one AP course (or scoring 3 or 
higher on at least one AP exam) for the comparison schools. yoi is the main effect of the 
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treatment, which captures the difference in log-odds of taking at least one AP course (or scoring 
3 or higher on at least one AP exam) between the comparison and the CRP schools. 


b. 2-Level HGLM With a School-Level Treatment Indicator and the Prior Year’s AP 
Exam Performance as a Covariate 


The level 1 model is the same as the above model (1a). Compared with 1b, the level 2 
model was extended by including the school-level covariate, pr2016x. This covariate was defined 
as the percentage of students in the 2015—2016 school year who took an AP course (or scored 3 
or higher on an AP exam) in an equivalent subject category as the 2017 AP data. The parameter 
yo1 in Equation 2a captures the difference in log-odds of taking at least one AP course (or scored 
3 or higher on an AP exam) between the comparison and the CRP schools, controlling for the 


effect of the covariate. The extended level 2 model was specified as: 


Tok = yoo + yorTrtk + yo2zpr2016x + rox rox ~ N(O, vo) (2a) 


Table 8 
Outcomes and Covariate in 2-Level HGLM 


Covariate 

Outcome (% of students in school in 2015—2016 school year) 
Math course taking Math course taking, 2016 
Science course taking Science course taking, 2016 
Math or science course taking Math or science course taking, 2016 
English course taking English course taking, 2016 
Math, science, or English course taking Math, science, or English course taking, 2016 
Any AP course taking Any AP course taking, 2016 
Math: AP exam score 3 or higher Math: AP exam score 3 or higher, 2016 
Science: AP exam score 3 or higher Science: AP exam score 3 or higher, 2016 
Math or science: AP exam score 3 or higher Math or science: AP exam score 3 or higher, 2016 
English: AP exam score 3 or higher English: AP exam score 3 or higher, 2016 
Math, science, or English: AP exam score 3 or Math, science, or English: AP exam score 3 or higher, 
higher 2016 
Any AP exam scored 3 or higher Any AP exam scored 3 or higher, 2016 


C. Results from 2-Level Hierarchical Generalized Linear Model 


a. Likelihood of Students Taking an AP Course 


We first present the 2-level HGLM results where the outcome is whether or not a student 


took at least one AP course in a subject category. Table 9 reports the estimates (in logit scale) of 
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all the fixed effects, p values, and the associated predicted probabilities of students taking at least 
one AP course for each subject category. As noted above, yoo represents the average log-odds of 
students taking at least one AP course for the delayed treatment schools and yo is the main effect 
of the treatment, which captures the difference in the log-odds of students taking at least one AP 
course between the study conditions. Thus, the linear combination of yoo+ yo1 represents the 
average log-odds of students taking at least one AP course for the treatment schools. The 
predicted probabilities are calculated from the predicted logit of the fixed estimates given the 


random effect rox is 0. 


For math, the estimate of intercept yoo was -2.75 with the p value smaller than 0.05, 
indicating that the expected log-odds of students in the delayed treatment schools taking at least 
one math AP course was -2.75 which is significantly different from 0. This estimate corresponds 
to a probability of 1/(1+exp[2.75]) = 0.06. The estimate of treatment effect (yo1 = 0.18, p value = 
0.57) was positive but was not statistically significant at the 0.05 level. This implies that the 
expected log-odds of students in the treatment schools taking at least one math AP course 
was -2.75 + 0.18 = -2.57 corresponding to a predicted probability of 1/(1+exp[2.57])=0.07. 
However, the difference in the log-odds of taking a math AP course between the study conditions 


was not significant. 


Similar results were obtained for science. The estimate of intercept yoo = -2.30 (p value < 
0.05) and the estimate of the treatment effect yo: = 0.17 (p value = 0.73). These results indicate 
that the expected log-odds of students in the delayed treatment schools taking at least one science 
AP course was -2.30 corresponding to a probability of 0.09 and the log-odds of students in the 
treatment schools was -2.30 + 0.17 = -2.13 (the predicted probability = 0.11). These results 
suggest that the difference in log-odds of students taking a science AP course between the study 


conditions is not statistically significant at the 0.05 level. 


Likewise, the results for English showed that the estimate of intercept was -1.88, and the 
estimate of the treatment effect was 0.33 (p value = 0.23). These results indicate that the 
expected probabilities of taking at least one English AP course in the delayed treatment schools 


and the treatment schools were, respectively, 0.13 and 0.17. 


As would be expected, similar results were found both in the math, science, or English 
category as well as for any AP courses. The treatment effects for these subject categories were 
not statistically significant at the 0.05 level (yoi = 0.41, p value = 0.3 for Math or Science; yoi = 
0.11, p value = 0.67 for math, science, or English; yo: = 0.42, p value = 0.21 for all AP exams). 


Overall, being in the schools which implemented CRP corresponded with a more positive 


log-odds estimate of taking at least one AP course for all subject categories. Despite the positive 
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values of fixed effect estimates of yoi, the results indicate that the effectiveness of CRP was not 
statistically significant at the 0.05 level. These results were also supported by the small 
increments in the predicted probabilities between the delayed treatment and treatment schools, 


ranging from 0.01% in math and 0.1% in any AP courses. 


Table 9 
2-Level HGLM Results: AP Course Taking as Outcome 


Outcome Fixed effect Estimate p value Predicted probability 
Yoo, Intercept -2.75 0.00 0.06 
Math 
You, Trt 0.18 0.57 0.07 
Yoo, Intercept -2.30 0.00 0.09 
Science 
you, Trt 0.17 0.73 0.11 
Yoo, Intercept -1.85 0.00 0.14 
Math or science 
You, Trt 0.41 0.30 0.19 
Yoo, Intercept -1.88 0.00 0.13 
English 
you, Trt 0.33 0.23 0.17 
Yoo, Intercept -1.17 0.00 0.24 
Math, science, or English 
You, Trt 0.11 0.67 0.26 
Yoo, Intercept -0.93 0.00 0.28 
All AP courses 
You, Trt 0.42 0.21 0.38 


b. Likelihood of Students Scoring 3 or Higher on an AP Exam 


Table 10 provides the results of 2-level HGLM with the outcome of the likelihood of 
scoring 3 or higher on an AP exam. For math, the estimate of intercept yoo = -4.00 (p value < 
0.05) represents the expected log-odds of students in the delayed treatment schools scoring 3 or 
higher on at least one math AP exam, and the predicted probability is 0.02. The estimate of 
treatment effect was yoi = - 0.53 (p value = 0.22), indicating that the log-odds of students in the 
treatment schools scoring 3 or higher was lower than the log-odds for the delayed treatment 
school by 0.53, corresponding to a decrease in the probability by 0.01. However, the decrease in 


the log-odds between the study conditions was not statistically significant at the 0.05 level. 


Similar results were obtained for science. The estimate of intercept yoo = -4.21 (p value < 
0.05) and the estimate of the treatment effect yo = -0.32 (p value = 0.49). These results indicate 
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that the expected log-odds of students scoring 3 or higher on at least one science AP exam 

was -4.21 and -4.53 for the delayed treatment and the treatment schools, respectively. Both 
estimates were associated with the same predicted probability of 0.01. These results suggest that 
the log-odds of students scoring 3 or higher on a science AP exam was lower for the treatment 
schools by 0.32, but the difference between the two groups was not statistically significant at the 
0.05 level. 


The results for English also showed the same trend. The estimates of the treatment effect 
were -3.50 (p value < 0.05) and -0.71 (p value = 0.14), respectively. The predicted probabilities 
for the delayed treatment and the treatment schools were 0.03 and 0.01, respectively. The 


treatment effect was not statistically significant at the 0.05 level. 


As expected, similar results were found when looking at the three combined subject 
categories. The treatment effects for these subject categories on the likelihood of students scoring 
3 or higher on an AP exam were not significant at the 0.05 level (yo1 = -0.33, p value = 0.42 for 
math or science; yoi = -0.64, p value = 0.12 for math, science or English; yoi = -0.45, p value = 
0.25 for any AP exams). 


Taken together, the results indicate that being in the treatment schools resulted in a slightly 
lower log-odds of a student scoring 3 or higher on at least one AP exam for every subject 
category. However, the estimate of the treatment effect was not statistically significant at the 
0.05 level. These were also confirmed by the negligible changes in the predicted probabilities 
between the treatment and delayed treatment schools, where the largest difference was 3% for all 
AP courses. The above findings from Table 8 and Table 9 agree with those from the descriptive 


Statistics in the previous section. 
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Table 10 
2-Level HGLM Results: AP Score of 3 or Higher as Outcome 


Predicted 
Outcome Fixed effect Estimate p value probability 
Yoo, Intercept -4.00 0.00 0.02 
Math: score 3 or higher 
You, Trt -0.53 0.22 0.01 
Yoo, Intercept -4.21 0.00 0.01 
Science: score 3 or higher 
You, Trt -0.32 0.49 0.01 
Yoo, Intercept -3.50 0.00 0.03 
Math or science: score 3 or higher 
You, Trt -0.33 0.42 0.02 
Yoo, Intercept -3.50 0.00 0.03 
English: score 3 or higher 
You, Trt -0.71 0.14 0.01 
Math, science, or English: score 3 00. Intercept -2.82 0.00 0.06 
or higher you, Trt -0.64 0.12 0.03 
Yoo, Intercept -2.66 0.00 0.07 
All AP courses: score 3 or higher 
You, Trt -0.45 0.25 0.04 


c. Likelihood of Students Taking an AP Course Controlling for Prior Year’s School- 
Level Performance 


To increase the precision of the treatment effect estimate, we further examined the 
treatment effect with the school-level AP performance in the previous year as a covariate. Table 
11 presents the results of these analyses. In general, we can see that the effects of covariate yo2 
were all positive and statistically significant at the 0.05 level (yo2 = 0.19 for math; yo2 = 0.10 for 
science; 0.07 for English; yo2 = 0.08 for the math or science and the math, science, or English 
categories; yo2 = 0.03 for any AP course with p values < 0.05). These estimates indicate that not 
surprisingly, the prior school-level AP performance had a significant association with the 
likelihood of students taking an AP course. 


Focusing on the main effect of the treatment yo1, being in the treatment schools did not 
have a statistically significant association with the log-odds of a student taking at least one AP 
course for every subject category, holding constant the effect of the 2016 school-level AP 
performance. Similar to those from the previous models, the estimates of treatment effect for 
math (yo1 = 0.07, p value = 0.74), science (yoi = 0.22, p value = 0.63), math or science (yo = 
0.40, p value = 0.25), and math, science, or English (yo: = 0.32, p value = 0.07) were positive 


values but not statistically significant at the 0.05 level. 
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Interestingly, however, the effectiveness of the CRP for English was found to be significant 
at the 0.05 level. Specifically, the estimate of treatment effect yo1 was 0.49, which was 
statistically significant at the 0.05 level. The predicted probabilities for the delayed treatment and 
the treatment schools were 6% and 10%, respectively, after controlling for the effect of last 


year’s school AP performance. 


Table 11 
2-Level HGLM Results: AP Course Taking as Outcome and 2016 AP Performance as Covariate 


Outcome Fixed effect Estimate —_—p value Predicted probability 

Yoo, Intercept -3.28 0.00 0.04 

Math You, Trt 0.07 0.74 0.04 
Yo2, pr2016 0.19 0.00 

Yoo, Intercept -2.81 0.00 0.06 

Natural science You, Trt 0.22 0.63 0.07 
Yo2, pr2016 0.10 0.00 

Yoo, Intercept -2.39 0.00 0.08 

Math or science You, Trt 0.40 0.25 0.12 
Yo2, pr2016 0.08 0.00 

Yoo, Intercept -2.69 0.00 0.06 

English You, Tit 0.49 0.03 0.10 
Yo2, pr2016 0.07 0.00 

Yoo, Intercept -2.17 0.00 0.10 

Math, science, or English You, Trt 0.32 0.07 0.14 
Yo2, pr2016 0.08 0.00 

Yoo, Intercept -1.48 0.00 0.18 

All AP courses You, Trt 0.49 0.11 0.27 
Yo2, pr2016 0.03 0.00 


d. Likelihood of Students Taking an AP Exam Controlling for Prior Year’s School- 
Level Performance 


Table 12 provided the fixed effect estimates, p values, and associated predicted 
probabilities of students scoring 3 or higher on at least one AP exam for each subject category 


after controlling for the effect of the 2016 school-level AP performance. 
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The treatment effects in Table 12 were not statistically significant holding constant the 
previous year’s school AP performance (yo = 0.66, p value = 0.08 for science; yoi = 0.10, p 
value = 0.77 for English; yo1 = 0.19, p value = 0.56 for math or science; yo1 = 0.07, p value = 
0.81 for math, science, or English). As with the results in Table 11, however, the effects of 
covariate yo2 were statistically significant at the 0.05 level (yo2 = 0.55 for math; yor = 1.08 for 
science; 0.43 for English; yo2 = 0.49 for the math or science category; yo2 = 0.39 for the math, 
science, or English category; yo2 = 0.14 for any AP exams with p values < 0.05). 


Table 12 
2-Level HGLM Results: AP Score of 3 or Higher as Outcome and 2016 AP Performance as Covariate 


Outcome Fixed effect Estimate pvalue Predicted probability 

Yoo, Intercept -4.59 0.00 0.01 
Math: score 3 or higher You, Trt -0.25 0.45 0.01 

Yo2, pr2016 0.55 0.00 

Yoo, Intercept -5.48 0.00 0.00 
Science: score 3 or higher You, Trt 0.66 0.08 0.01 

Yo2, pr2016 1.08 0.00 

Yoo, Intercept -4.44 0.00 0.01 
Math or science: score 3 or higher You, Trt 0.19 0.56 0.01 

Yor, pr2016 0.49 0.00 

Yoo, Intercept -5.02 0.00 0.01 
English: score 3 or higher You, Trt 0.10 0.77 0.01 

Yor, pr2016 0.43 0.00 

Yoo, Intercept -4.17 0.00 0.02 
Math, science, or English: score 3 or higher 1, Trt 0.07 0.81 0.02 

Yor, pr2016 0.39 0.00 

Yoo, Intercept -3.63 0.00 0.03 
All AP courses: score 3 or higher You, Trt -0.05 0.88 0.02 

Yo2, pr2016 0.14 0.00 


V. Results for Implementation Evaluation 


The fidelity matrix of implementation defines the key components of the CRP program 
depicted in the CRP logic model (see Figure 1), measures of each component, scoring rubrics of 
measures, and criteria of fidelity. Based on this fidelity matrix, implementation information was 


collected from administrative records, surveys, and interviews. 
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A. Fidelity Matrix 


The fidelity matrix approach collected information based on observable and measurable 
indicators relating to key program indicators. The CRP logic model posits that the key 
components of the intervention are school, teacher, and student supports. The idea was to 
measure fidelity separately for each key component of the intervention and define threshold 
values (in collaboration with NMSIJ) to determine whether the intervention was implemented 
with fidelity. As a starting point, we used fidelity indicators developed and field-tested (e.g., 
Sherman, et al., 2015), and created an implementation fidelity matrix which links the key 
components of the intervention to their indicators, the data source, the indicator scoring system, 


and the implementation threshold values (see Appendix A). 


Fidelity was measured separately for each key component of the intervention to determine 


whether the intervention was implemented with fidelity. 


In some cases implementation could be measured on a yes/no basis (e.g., did schools 
receive materials, or were exam fees paid by NMSI). These elements were considered to have 
been implemented with fidelity if in at least 80% of the schools they were implemented as 
planned. Some program elements required a number of individuals participating in an event (e.g., 
attending a summer training session, or attending three student study sessions). In these cases, if 
80% of identified staff or students attended, these elements were considered to have been 


implemented with fidelity. 


B. Surveys and Interviews 


We also collected implementation information from surveys and interviews. Interview and 
survey data collected for the study allow us to determine how components of the program 
function in “real-world contexts,” and provide support and validation for the fidelity matrix data. 
More importantly, these data help us learn how components of the program are viewed by those 
within the school, and how teachers can be supported so programs function effectively and lead 
to positive change. Obtaining teachers’ firsthand views and opinions provides unique insight into 
what is necessary to build and sustain an effective AP program and supportive school culture, 
and if need be, provide formative feedback to help guide modifications or enhancements. 
Teachers are more likely to understand the complexity of their particular school, classroom, and 


student population and are most closely connected to actual program implementation. 


To determine the perceived effectiveness of program elements we created an online survey 
and interview protocol. The survey focused on the key components of the CRP: teacher training, 
student supports, administrative support, monetary incentives, additional instruction (via 


Saturday sessions), classroom supplies/equipment, and change in school culture as relates to the 
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AP program. The online survey questions were used as a basis for the interview protocol and 
allowed us to expand on topics in the survey. The measure was based on one created for previous 
studies of the CRP (Cross, Kilpatrick, & LaMonica, 2012), with additional items added for this 
project. 


a. Online Surveys 


The online surveys were created using the web-based survey creation program Survey 
Monkey. Alternate versions were created for teachers and administrators. Prior to sending the 
surveys to participants we sent an introductory email explaining the research study, participation 
requirements, as well as the time frame for participation. Eligible participants were subsequently 
emailed a link to the survey with the study overview and an IRB-required consent form. Prior to 


the deadline, several reminder emails were sent. 


b. Teacher Survey 


The teacher survey contained 53 questions, although participants did not all answer every 
question. Skip logic was employed to allow teachers to move past a set of questions if they were 
not relevant or applicable. For example, item 21 asked teachers: “Did your school hold shared 
goal setting meetings to establish targets for student performance and enrollment in AP 
courses?” If the respondent’s school did not hold shared goal setting meetings, they skipped past 
the goal setting meeting questions. The teacher survey measure is included as Appendix B along 


with information on the skip logic employed throughout. 


The majority of survey questions were multiple choice, multiple select (respondents could 
choose more than one answer), or questions with a Likert scale (most often four-point). The first 
section of the survey included demographic questions (respondent’s school, courses taught, 
grade level etc.) as well as questions on professional development participation. The rest of the 
survey sections were aligned to the CRP logic model constructs and addressed school support (3 


items), teacher support (17 items) and student support (4 items). 


c. Administrator Survey 


The administrator survey was similarly constructed, albeit aligned to the administrative 
elements of the logic model. Thus, there were questions focused on coordination of the CRP, 
student supports, and administrator incentives, as well as identical questions to those on the 
teacher survey about effectiveness of program components on enrollment in AP courses and 


performance on AP exams. See Appendix C for the administrator survey measure. 


21 


d. Student Survey 


Unlike the teacher and administrator surveys, the student surveys were paper booklets 
designed to be completed by hand in class. Following completion of the online teacher survey, a 
packet of student surveys specifically coded for the teacher and school were sent to the teacher 
with a return envelope. The survey consisted of 29 questions about the student’s experience with 
the CRP and AP courses in general. 


The survey began with five questions about the student’s current AP workload, college 
aspirations, and the school’s outreach efforts to students concerning AP course enrollment. The 
students were then asked 15 questions about the supports available to them, not limited to CRP 


components, including six questions intended to solicit qualitative assessments of the supports. 


The survey continued with five questions about the student’s perception of their own 
preparedness and potential obstacles to students taking their AP exams. The final four questions 
identified the various incentives offered to students and gauged the student’s valuation of them. 
The incentives probed included CRP financial awards and other perceived benefits, such as 


weighted calculations of grade point average. See Appendix D for the student survey measure. 


e. Teacher and Administrator Interviews 


As a supplement to the surveys we created a set of teacher and administrator interview 
questions to provide more detailed information which may not have been gleaned from the 
online survey. The survey questions formed the basis of interviews with teachers and 
administrators, and additional questions were included as dictated by the nature of the 
conversation. We did not want to constrain teachers and so allowed the discussions to evolve as 
they progressed. Appendix E and Appendix F provide the initial set of interview questions for 


teachers and administrators. 


C. Fidelity Matrix Results 


The fidelity matrix relates to all components of the CRP, includes targets for each element 
of the program and requires 80% of schools meet specified targets. Incomplete data made 
completing the matrix in the way intended challenging. We did not receive administrator surveys 
from 12 schools, student surveys from five schools, and had incomplete student attendance 
records for the student study sessions. Data to complete the matrix was gathered from 


administrative records as well as survey and interview responses where necessary. 


a. School-Level Indicators 


School-level indicators included program management support, school-wide goal setting, 


and payment of administrator awards. Complete data were available for 16 schools or 57% of the 
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possible sample (and incomplete data for 12 schools). School-level indicators were all evaluated 


on a yes/no basis. 


Seven specific school support measures were identified answering four key questions: 


e Did NMSI and school personnel meet to set goals for the 2016-2017 school year? 


e Were stipends paid to school administrators, as agreed? 


e If goals were met, were Designated Administrator bonuses paid? 


e Did Program Managers support the school by delivering student study session reminders 
and materials, assisting with teacher sign-ups for training and student study sessions, 
assigning mentors to AP teachers, and providing guidance to NMSI curricular support 


materials? 


School personnel in 25 of the 28 treatment schools (89.3%) confirmed that a goal setting 


meeting took place. NMSI financial records indicated that the school administrator stipend and 


bonuses were paid to all qualifying schools. The Program Manager evaluation data were only 


obtained through the 16 completed Designated Administrator surveys. “Yes” responses to these 


survey items ranged from 87.5% (mentor assignment) to 100.0% (teacher sign-ups). Table 13 


summarizes the school support measures across all schools and reveals that each program 


element was implemented with fidelity. 


Table 13 


School-Level Implementation Indicators 


School-level indicator # of responses Yes responses % Yes 
Goal Meeting Held 28 25 89.3 
Administrator Stipend Paid 28 28 100.0 
Administrator Bonus Paid According to Agreement 28 28 100.0 
Program Manager Support: Teacher Sign-Ups 16 16 100.0 
Program Manager Support: Student Study Sessions 16 15 93.8 
Program Manager Support: NMSI Materials 16 15 93.8 
Program Manager Support: Mentor Assignment 16 14 87.5 


b. Teacher-Level Indicators 


Eight specific metrics were identified to evaluate teacher participation in the CRP and 


NMSI support of teachers. Eleven schools (39%) achieved a perfect score on the teacher support 


program elements. Table 14 summarizes the range of scores in the teacher measures. 
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Table 14 


Measures of Teacher Participation and Support 


Type Score of 6 Score of 7 Score of 8 
# of schools 3 14 11 
% of schools 10.7% 50% 39.3% 


The CRP supports teacher instruction by providing materials for the classroom, online 
access to instructional materials, mentoring, and off-site training. Financial records indicated that 
each school received funding to purchase school room materials and supplies, ranging from 
books to lab supplies. Teachers in 24 of the schools (85.7%) responded affirmatively when asked 
if they had the materials they need to teach their course effectively. Teachers in all 28 schools 
were made aware of the availability of online instructional resources and logged into the system 


at least once. Mentors were also offered to teachers in all 28 schools. 


Schools were evaluated on the level of teacher participation in CRP training sessions by 
assessing whether teachers attended all three sessions available throughout the year (Advanced 
Placement Summer Institute, Fall Workshop, and Spring Training). If, in the aggregate, teachers 
attended 80% or more of the available training sessions, the school received a score of one and 
this was achieved by 13 schools (46.4%). Some teachers cited the weeklong commitment as an 
obstacle to attending the summer training, whereas other roles and responsibilities that teachers 
maintain (e.g., coaching, advising student organizations) can also make attendance during the 


school year challenging. Five schools (17.8%) had 100% teacher attendance at training sessions. 


The CRP also offers financial support to teachers for participating in the program, both 
through a stipend and through awards tied to achieving goals. NMSI financial records confirmed 
that teachers at all 28 schools received stipend payments for program participation. For each 
student with an exam score of three or higher, teachers should have received an award of $100. 


Payments to teachers according to the agreement were verified for 27 schools (96.4%). 


Across all schools teachers did not fulfill their requirements for attending training sessions, 
and so this component was not implemented with fidelity. The other seven program elements 


were implemented with fidelity in the aggregate across all schools. 


c. Student Supports 


Data on student attendance at the Saturday study sessions was not complete and so no 
calculation of overall implementation fidelity was possible. For the students for whom we did 


receive data, however, the overall attendance across schools was 55%. Fee subsidies were 
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offered to students at 60.7% of schools which indicates not implementing with fidelity. There 
may, however, have been instances of students receiving fee subsidies from other sources of 
which we were not aware. Students were promised a $100 award for each exam with a score of 3 
or better. NMSI financial records confirm that payments were made as expected in 27 of the 28 
schools (96.4%), indicating that this program element was implemented with fidelity. Out of a 
maximum aggregate score of two for the student measures, 12 schools scored the maximum 


(42.9%) and 16 schools achieved a score of one (57.1%). 
d. Overall School-Level Fidelity 


Each of the measures was tallied per school, and the school’s total score was evaluated as a 
percentage of the maximum points available at that school. Twenty-three schools (82.1%) 
achieved 80% or better implementation fidelity, for an average fidelity score of 89.3%. Seven 
schools achieved a perfect 100% fidelity score. Table 15 represents the overall school-level 
fidelity. 
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Table 15 
Overall School-Level Fidelity 


School Implementation fidelity score 
CA school 1 94.1 
CA school 2 88.2 
GA school 1 76.9 
GA school 2 100.0 
IL school 2 92.3 
IL school 3 82.4 
IL school 1 100.0 
LA school 1 88.2 
MI school 1 92.3 
MI school 2 92.3 
MO school 1 94.1 
MO school 2 100.0 
MO school 3 94.1 
ND school 1 92.3 
ND school 2 94.1 
ND school 3 92.3 
OH school 3 100.0 
OH school 1 82.4 
OH school 2 82.4 
OH school 4 84.6 
PA school 1 100.0 
PA school 2 100.0 
PA school 3 100.0 
TX school | 76.9 
TX school 3 69.2 
TX school 4 84.6 
TX school 2 69.2 
TX school 5 76.9 


In the aggregate by component, the percentage of schools that adequately implemented the 


measures ranges from 57.1% for the student-level measures to 100% for the school measure 
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(complete data). Table 16 delineates by component the number and percentage of schools 


performing at or above 80% fidelity. 


Table 16 
Schools Adequately Implementing Aggregate Measures by Component 


# adequately 
implemented (at or above % adequately 
Component # of schools 80%) implemented 
School-level indicators (complete data) 16 16 100.0 
School-level indicators (incomplete data) 12 10 83.3 
Teacher-level indicators 28 25 89.3 
Student-level indicators 28 16 57.1 


D. Survey Results 


The research objectives for this part of the evaluation study were to better understand 
teacher and administrator perspectives on the effectiveness or impact of key CRP components on 
student interest and success in AP and school culture. For all survey questions, frequencies for 
each response category were first calculated. Some questions required respondents to respond 
using Likert (4 or 3 points) or dichotomous scales. For these questions we calculated mean 
values to gain an understanding of average level of agreement with the statements. Some 
questions shared a common prompt. For these questions, we further computed the average of the 


set of item responses as a composite score per respondent and obtained descriptive statistics. 


a. Response Rates 


Completed surveys were received from 129 teachers (around 78% of the eligible sample). 
This represents a high level of participation, perhaps influenced by incentives offered to 
respondents (a $50 gift card) as well as pre-notice contact and several email reminder messages. 
We received a teacher survey from at least one teacher from each of the 28 schools in the 
sample, with a 100% response rate from 13 schools. Administrator surveys were received from 
16 administrators. Table 17 provides the distributions of teachers responding to the survey by 


state, district, and school. 
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Table 17 


Distribution of Teacher Survey Respondents by State, District, and School 


State District School n % 
CA school 1 3 2.33 
CA CA district 
CA school 2 5 3.88 
GA school 1 6 4.65 
GA GA district 
GA school 2 5 3.88 
IL school 1 2: 1.55 
IL IL district IL school 2 3 2.33 
IL school 3 4 3.1 
LA LA district LA school 1 7 5.43 
MI school 1 9 6.98 
MI MI district 
MI school 2 iL 5.43 
MO school 2 4 3.1 
MO MO district MO school 1 4 3.1 
MO school 3 7 5.43 
ND school 1 4 3.1 
ND district 1 
ND ND school 2 8 6.2 
ND district 2 ND school 3 5 3.88 
OH school 2 1 0.78 
OH school 1 2 1.55 
OH OH district 
OH school 3 3 2.33 
OH school 4 2 1.55 
PA district 1 PA school 1 4 3.1 
PA PA district 2 PA school 2 8 6.2 
PA district 3 PA school 3 7 5.43 
TX school 3 2 1.55 
TX school | 1 0.78 
TX TX district TX school 4 4 3.1 
TX school 2 6 4.65 
TX school 5 6 4.65 


b. Teacher Survey Response Summary 


Over 90% of teacher respondents had three or more years of teaching experience. Fifty- 


nine teachers had a single subject credential and 43 multiple subject credentials. The majority of 
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respondents taught 12" grade (n = 108) and/or 11" grade (n = 106). Sixty-nine teachers taught 
10 grade and 39 teachers ninth grade. On average, participants had been teaching for 13.06 
years (SD = 8.36, Mdn = 12.00), and had been teaching AP courses an average of 4.75 years (SD 
= 5.36, Mdn = 3.00). 


We divide the results from the surveys into teacher perceptions and opinions on the effects 
of the three categories of CRP components: (1) teacher, (2) student, and (3) school-focused on 
teacher knowledge and pedagogy, student outcomes, as well as student interest and participation 
in AP. Key teacher support components were training, mentoring, incentives, and resources. 


c. Teacher Components 


1. Training 


Teachers participated in three training events: a four-day summer training, a two-day fall 
training and a one-day spring training. We asked teachers which of a set of statements was true 
for each of the three training sessions they attended. Specifically we were interested in how 
effective teachers found the training sessions for achieving stated goals. Figure 4 depicts the 
results of these survey items. In all cases, fewer teachers found the training sessions effective in 
helping them differentiate instruction for different ability level students (between 43% and 50%). 
The number of teachers, however, who felt more qualified to be an effective AP instructor 
increased throughout the year. After the summer training, 67% felt more qualified, after the fall 
training this increased to 71% and following the spring training, over three quarters of teachers 
felt more qualified to teach AP. 
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Figure 4. Teacher ratings of training efficacy. 
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We also asked teachers to indicate how much the CRP had improved their content 
knowledge, instructional skills, techniques and strategies. Half of the respondents felt that the 
CRP contributed to a major improvement in content knowledge (average rating was 2.48 on a 
three-point scale: 3 = major improvement, 2 = slight improvement, 1 = no improvement). 
Similarly, just over half of the respondents indicated improvement in their instructional skills and 
strategies following the CRP (average rating was 2.48). 


The student-study sessions offered opportunities for teachers to observe expert teachers and 
learn new instructional techniques or strategies. We asked teachers to rate the extent of their 
agreement on a four-point scale with a set of statements about the student study sessions. For this 
set of questions the scale was 4 = strongly agree, 3 = agree somewhat, 2 = disagree somewhat, 
and 1 = strongly disagree. A separate item asked teachers to indicate how useful the sessions 
were for them. For this set of questions, the four-point scale was 4 = extremely useful, 3 = useful, 
2 = somewhat useful, and 1 = not at all useful. Table 18 presents results for these questions 
combined together. Teachers’ level of agreement for all statements was similar—ranging from 


an average of 2.94 to 3.06. 


Table 18 
Teacher Ratings of Student Study Session Efficacy 


Student study session statements n M SD 


The study sessions highlighted the instructional needs of the students so I 


could continue to address them in class. i 208 Oe? 
I learned a great deal from watching the expert teachers. 125 2.94 0.92 


I was able to take the strategies employed during the study sessions back 


to the classroom to help improve student achievement. ee ae vee 


How useful were study sessions for you? 124 3.02 0.93 


Over 70% of teachers indicated the student study sessions were useful or extremely useful. 
Similarly, teachers expressed consistent views (over 70% of teachers reported somewhat or 
strong agreement) on the efficacy of the specific components of the study sessions shown in 
Table 18. 


2. Mentoring 


Teachers were asked if mentoring was offered to them through the CRP and 113 teachers 
(87.6%) indicated it was. Of this group, 43 teachers indicated they did not have any contact with 


the mentor. We asked the remaining group of teachers the extent of their agreement (on a four- 
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point scale, where 4 = strongly agree, 3 = agree somewhat, 2 = disagree somewhat, and | = 
strongly disagree) with a set of statements aligned to the mentoring objectives. Table 19 presents 
results for the mentoring-related questions. Teachers had the highest agreement relating to the 
preparedness of their mentor (VM = 3.7, SD = .61), followed by agreement that the mentor helped 
improve content knowledge (M = 3.22, SD = .92). The most commonly provided support by 
mentors was information on pacing and sequencing of lessons (71%), with the least commonly 


provided support direct instruction or tutoring for CRP students (24%). 


Table 19 


Teacher Opinions of Mentoring 


Mentoring-related statements n Mean SD 
The mentor was well-prepared 63 3.7 0.61 
The mentor improved my content knowledge 63 3.22 0.92 
The mentor honed my skills and techniques 61 3.1 0.94 
Because of mentoring I am more effective 61 3.07 0.91 


Note. Scale is 1 = Strongly disagree, 2 = Disagree somewhat, 3 = Agree somewhat, 4 = Strongly agree. 
3. Incentives 


Eighty-four percent of teachers (n = 107) indicated they were offered incentives (financial 
or otherwise) for teaching AP courses through the CRP. Almost all of these teachers stated they 
were offered financial incentives, with 58% indicating they were offered professional 
development opportunities and 6% were offered recertification points. Incentives were most 
commonly offered for students passing the AP exam (89%). Thirty-five percent of teachers said 
the incentives were not at all important in encouraging them to teach AP courses, with an 
additional 35% indicating the incentives were somewhat important. Only 8% of teachers 
reported the incentives to be an extremely important component in encouraging them to teach in 


the AP program. 


4. Online Curricular Resources 


We asked teachers which additional resources were offered for use in their AP courses. 
Sixty percent of teachers indicated they were offered access to online resources, and 65% of 
teachers indicated they were offered use of AP curriculum materials from the College Board. 
Twelve percent of teachers said they accessed the online resources daily, with one third 
accessing them weekly and one third accessing them monthly. Fifteen teachers indicated they did 
not access the online materials at all. Teachers who reported accessing CRP online materials (n = 


111) most commonly did so to help familiarize students with the types of questions on the AP 
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exam (83%). The next most commonly selected use of materials was deepening instruction in 
specific content areas (71%) and conducting practice exams (70%). The least frequently chosen 


response was preparing for the student study sessions which was chosen by 23% of teachers (see 
Figure 5). 
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Figure 5. Teacher use of online resources. 


5. Goal Setting 


Forty-three percent of teachers reported their school held a goal-setting meeting, and of 
those teachers, 75% attended. The purpose of these meetings is to establish school-wide goals for 
enrollment in AP as well as student performance. Table 20 presents results from the goal setting 
meeting questions. Average agreement of meeting efficacy was 2.88 (SD = 0.71), with the 
highest average agreement for establishing goals for equitable access to AP coursework (M = 
3.17, SD = 0.70). 
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Table 20 


Teacher Perspectives on Goal Setting Meeting Impact and Efficacy 


Statement n Mean SD 

Shared goal setting meeting was effective. 41 2.88 0.71 
Shared goal setting meeting established enrollment goals. 40 3.05 0.64 
Shared goal setting meeting established exam performance goals. 41 2.95 0.74 
Shared goal setting meeting established goals for underrepresented 

: 41 3.10 0.80 
students in AP courses. 
Shared goal setting meeting established goals for equitable access to AP Al 3.17 0.70 


coursework. 


Note. Scale is 1 = Strongly disagree, 2 = Disagree somewhat, 3 = Agree somewhat, and 4 = Strongly agree. 
d. Overall Impact of the College Readiness Program on Enrollment and Achievement 


Teachers were asked their opinions on the overall effectiveness of four program elements 
as they related to having an impact on student enrollment. The program elements were student 
financial incentives, teacher financial incentives, open enrollment, and teacher training. 
Specifically which of the four components was seen as the most and least effective for increasing 
student enrollment in AP courses. For the same four plus an additional two program components 
(online instructional resources and student study sessions) teachers were asked to select which 
component was the most and least effective at increasing student performance. Other survey 


items also included questions relating to student enrollment and achievement in AP courses. 


1. Student Enrollment in AP Courses 


Thirty-six percent of teachers (n = 129) chose student financial incentives as the most 
effective component of the CRP at increasing student enrollment in AP courses. The second 
most commonly selected response was teacher training (chosen by 32% of teachers). Teacher 
incentives were viewed as the most effective component influencing student enrollment by only 
6% of teachers. We also asked teachers to indicate the least effective component of the CRP for 
increasing student enrollment. Forty-three percent of teachers chose the open enrollment policy 
as the least effective at increasing student enrollment, and 19% chose teacher incentives as the 


least effective component (see Figure 6). 
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Figure 6. Teacher ratings for which elements of the CRP were most and least effective at increasing student 
enrollment in AP courses. 


We asked teachers a follow-up question focusing on open enrollment specifically. Of the 
responding teachers (n = 115), 54% said that open enrollment had a positive impact on the AP 
program at their school. So even though teachers did not rank open enrollment as high as other 
program elements in terms of increasing enrollment, they did see it having an impact on the AP 
program as a whole—albeit not the most significant elemement compared to the other options. 
Teachers were also asked to indicate if, overall, the CRP was an effective way to increase student 
enrollment in AP courses. Seventy-four percent of teachers agreed the program did achieve this 
goal. Teachers were able to provide additional comments for this question. For those who did so 
(n = 18), some indicated that students were already encouraged to take AP courses at their 
school, others expressed concern that students be aware of the rigor associated with AP courses, 
and some teachers indicated it was too early to tell if there had been any impact. Twenty-nine 
percent of teachers agreed that the CRP contributed to a major improvement in recruitment of 
high-need and traditionally underrepresented students into AP course, and 57% of teachers felt it 


contributed to a slight improvement in this area. 


To further explore factors relating to enrollment, we asked teachers to indicate reasons why 
qualified students may not have enrolled in AP courses in the past. The most common reason 
was the courses have a reputation as being difficult (68% of responses). Other reasons chosen 
included: students not realizing they were qualified to take AP courses (28% of responses), lack 
of outreach (20%), and inability to afford the exam fees (18%). Twenty-nine teachers provided 
additional comments for this question, citing student apprehension at the increased work load, 
concern about jeopardizing their GPA, lack of general interest, and scheduling difficulties as 


common reasons students have previously not enrolled in AP courses. 
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2. Student Performance in AP Courses 


Thirty-six percent of teachers chose teacher training as the most effective component of the 
CRP at increasing student performance in AP courses and on AP exams, and one third of 
teachers chose the student study sessions as the most effective component. Only 16% of teachers 
chose student financial incentives as the most effective element (compared to 36% who saw 
them as the most important component for enrollment). Teacher incentives were chosen as the 
most important component for increasing student performance by only two teachers. Open 
enrollment was the most commonly selected component seen as the least effective at increasing 
student performance (45% of teachers), followed by teacher incentives, chosen as the least 
important component by 15% of teachers (see Figure 7 and Figure 8). 


70 
60 
-_ 36% 
Q 50 33% 
7 
= 40 
"i 
© 30 
Ss 16% 
© 
20 
5% 
10 2% 2% 
zs - = 
Teacher Student Student Online Open Teacher 
Training Study Financial Instructional Enrollment Incentives 


Sessions Incentives Resources 


Figure 7. Teacher ratings for which program element was the most effective at increasing student performance. 
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Figure 8. Teacher ratings for which program element was the least effective at increasing student performance. 
e. Administrator Survey Results 


Next we present findings from the administrator surveys. CRP administrators were asked 
their opinions on the effectiveness of elements of the CRP, specifically, (a) which element was 
seen as the most effective for increasing student enrollment in AP courses, and (b) which was the 
most effective at increasing student performance. Other survey items also included questions 


relating to both student enrollment and achievement in AP courses. 
1. Overall Impact of CRP 


a) Student Enrollment 


Forty-four percent of surveyed administrators (n = 16) saw open enrollment as being the 
most effective program element related to student participation in AP courses (compared to 22% 
of teachers), followed by 31% who chose student incentives (compared to 36% of teachers). 
Teacher incentives were seen as the least effective element relating to student enrollment by 44% 
of the administrators (compared to 19% of teachers), and teacher training was the second most 
commonly selected element (chosen by 25% of respondents compared to 16% of teachers). The 
similarity between teachers and administrators was only in the view of student incentives as 
being one of the top two most effective program elements to encourage student enrollment. 
Interestingly, the number one choice for administrators was open enrollment, but this was the 
most frequently selected element (by teachers) as being the least effective element relating to 


increasing enrollment (see Figure 9). 
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Figure 9. Administrator perspectives on which elements of the CRP were most and least effective at increasing 
student enrollment in AP courses. 


A follow-up question focused specifically on open enrollment. Eighty-eight percent of 
administrators (14/16) said open enrollment had a positive impact on the AP program at their 
school, and 88% also indicated that the CRP was an effective way to increase student enrollment 
in AP courses. Respondents were able to provide additional comments for this question. For 
those who did so (n = 6), four indicated that the training, study sessions, and/or the student 
incentives were important components. One indicated that the CRP simply served as a 
supplemental resource for an existing AP program, and one administrator reported that the CRP 
could be seen as an effective way to increase AP enrollment, as long as the students were aware 
of the incentives. Sixty-three percent of respondents agreed that the CRP contributed to a major 
improvement in recruitment of high-need and traditionally underrepresented students into AP 


courses, and 25% felt it contributed to a slight improvement in this area. 


b) Student Performance in AP Courses 


Figure 10 shows results from part (b) which component was the most effective at 
increasing student performance. There was little variance in response for the most effective 
performance-related component: Fifty-six percent of administrators (n = 9) chose the student 
study sessions as the most important component relating to student achievement (compared to 
33% of teachers), and 44% chose the teacher training as the most important (compared to 36% of 
teachers). Open enrollment was chosen by 38% of administrators as being the least effective 
(compared to 45% of teachers), with teacher incentives chosen by 25% (compared to 16% of 
teachers) (see Figure 10). 
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Figure 10. Administrator perspectives on which elements of the CRP were most and least effective at increasing 
student performance in AP courses. 


An additional set of questions asked administrators to indicate the extent to which the CRP 
contributed to improvement in certain areas (see Table 21). The highest level of perceived 
improvement was in teachers’ instructional skills, techniques, and strategies (VM = 2.63, SD = 
.72), followed by students’ content knowledge (M = 2.56, SD = 0.63). Seventy-five percent of 
respondents (n = 16) thought that the CRP contributed to a major improvement in teachers’ 
instructional skills, and 63% indicated major improvement in students’ content knowledge. 
Students’ experience with STEM was rated as less impacted by CRP, with 56% of respondents 
indicating a slight improvement (M = 2.19, SD = 0.66). In all cases, however, the average impact 
was at least slight improvement. 


Table 21 


Administrator Perceptions of CRP-Related Improvement (n = 16) 


Statement n Mean SD 
Students’ content knowledge 16 2.56 0.63 
Students experience with STEM AP courses 16 2.19 0.66 
Recruitment of high-need and traditionally underrepresented 16 2.50 0.73 
students into AP courses 
Teachers’ content knowledge 16 2.50 0.63 
Teachers’ instructional skills, techniques and strategies 16 2.63 0.72 
School culture of continuous improvement 16 2.38 0.72 
School leadership valuing STEM learning 16 2,25 0.69 


Note. Scale is 1 = No improvement, 2 = Slight improvement, and 3 = Major improvement. 
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f. Teacher Interviews 


We interviewed 47 teachers from 17 schools in eight states as well as six administrators 
from six schools. This group of teachers represents a convenience sample of schools and teachers 
based on availability and scheduling constraints. Most teachers were interviewed at their school, 
and some were interviewed during their attendance at a summer training session in August 2017. 
Interview questions were drawn from and aligned to key themes of the online survey, but were 


open ended to encourage less restricted conversation. 


1. Overall Efficacy of the CRP 


We first asked teachers their opinions on the most important or effective component of the 
CRP. Forty-five percent of the teacher interviewee sample (n = 47) indicated the training was the 
most effective component, which mirrored the findings from the teacher surveys. The second 
most frequently stated component was resources (12.8%) followed by student study sessions 
(9%). Teachers often included additional effective elements in their responses. Considering all 
components mentioned as effective, 75% of teachers indicated the training and 62% referred to 
resources as effective components. All but one administrator listed the training as an effective 
component of the program. In addition to training, student study sessions, and resources, 
administrators also mentioned the funds for school supplies and equipment as an important part 


of the program. 


Sixty-six percent of teachers and administrators felt that school culture had changed since 
the implementation of the CRP. However, almost half of the interviewees who felt the culture 
had not changed (7 of 15) said that it was most likely too soon to tell what impact the program 
would have on the school culture. 


2. Training and Support 


The majority of teachers (89%) were satisfied with the level of training and support 
received throughout the academic year as part of the CRP. Seventy-seven percent of teachers 
attended all three training sessions, and 98% attended at least one. For those who attended, 81% 
found the sessions useful. All but one teacher indicated they felt adequately prepared to teach 
their AP courses. Some administrators conducted classroom observations. These led one 
administrator to comment that instruction was more “clear,” and another noted that “teaching 


practices were enhanced.” 


3. Mentoring 


Ninety-six percent of teachers were offered a mentor, but only just over half of those 


teachers took advantage of the opportunity to meet with the mentor. The most common reason 
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teachers did not establish a mentee relationship was not having enough time (33%). Of those 
who met with the mentor, most (82%) found it beneficial. The two mentor activities most 
frequently cited as helpful were responding to questions from the teacher (noted by 74% of 


mentees) and providing supplemental resources to the teacher (33%). 


4. Incentives 


Teachers and administrators were about evenly split on how effective the student incentives 
were in terms of increasing student interest and enrollment in AP courses. The most frequently 
observed student motives for taken the AP courses were gaining college credit for the course and 


the benefit to the students’ weighted high school grade point average. 


Ninety-six percent of teachers indicated they were offered incentives to participate in the 
CRP, but most reported the incentives did not play a significant role in encouraging them to 
teach AP courses; 65% of respondents said the incentives were nice to have, but they did not 
factor into the decision to teach the course; and 30% of respondents indicated they were not 
motivating at all. Eighty percent of the administrators who discussed the program goals for 
students scoring 3 or higher on the exam felt some of the expectations for student performance 


were unrealistic. 


We asked teachers if they would change anything about the CRP to help improve AP 
education at their school. The top two changes were improved communication with NMSI (17%) 
and changing the student study session scheduling (also 17%). The latter issue also emerged as a 
theme across many of the interviews. The scheduling of the study sessions on Saturdays posed 
significant problems for many students because of work and sports team commitments as well as 
other family-related obligations. Additionally, some students who rely on public transportation 
encountered issues with different weekend schedules, passes that work on weekdays only, and 
finding transportation to a different school. Some teachers also indicated that they would like 
more emphasis on test-taking strategies, and some felt that the student incentives should be 


awarded for attending the study sessions rather than achieving a particular score on the AP exam. 


g. Student Surveys 


Below we summarize data from the 877 students who responded to the CRP student survey 
(see Table 22 for distribution of respondents). Surveys were sent to teachers indicating interest 


and students completed paper-and-pencil copies which were then mailed back. 
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Table 22 


Distribution of Student Survey Respondents by State and District 


State District n % 
CA CA district 98 11.17 
GA GA district 49 5.59 
IL IL district 36 4.1 
LA LA district 53 6.04 
MI MI district 116 13.23 
MO MO district 81 9.24 
ND ND district 1 89 10.15 
OH OH district 72 8.21 
PA PA district 1 26 2.96 

PA district 2 49 5.59 
PA district 3 38 4.33 
TX TX district 170 19.38 


1. AP Courses 


a) Number of AP Courses 


The average number of math, science, and English AP courses taken by all students in the 
sample was 2.82 (SD = 1.82, Mdn = 2.00), with a range from 1-15. 


b) AP Course Subjects 


Figure 11 shows the number of specific AP courses students reported taking during 2016— 
2017. The largest number of respondents were taking English language (n = 472; 54%), followed 
by physics (n = 270; 31%). 
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Figure 11. Number of AP courses taken by students in the survey study sample. 


2. Knowledge of AP and the CRP 


We asked students how they learned about the AP program in their school as well as how 
they learned about the CRP specifically. Most students learned about AP and the CRP from their 
teachers or school counselors, and more than a third of students learned about the AP courses 
from other students (see Table 23). In terms of learning about AP courses, responses in the 
“other” category included parents and family members, elementary/middle school teachers, 
NMSI, and the gifted coordinator. Some students indicated that they were just enrolled in the 
class, it was mandatory to enroll, or they were selected. When asked how they learned about the 
CRP, 22 of the students who responded “other” indicated that the survey was the first time they 
learned the name of the program. 
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Table 23 
Sources of Student Knowledge of the AP and College Readiness Program 


Source Learn about AP Learn about CRP 
My AP Teachers 522 (59.5%) 635 (72.4%) 
Other teachers at school 350 (39.9%) 138 (15.7%) 
School counselor 408 (46.5%) 279 (31.8%) 
Other students 333 (38%) 110 (12.5%) 
School signs, emails, fliers 87 (9.9%) 47 (5.4%) 
People outside school 39* (4.4%) 11 1.3%) 
Other 35 (4.0%) 31 3.5%) 


3. Future Educational Plans 


Ninety-five percent of student respondents (n = 877) indicated they planned to attend some 
sort of postsecondary institution. Of these the highest level of education students planned to 
complete was two-year community/junior college (3%), four-year college/university (46%), and 
graduate school (46%). 


4. Tutoring 


Seventy-two percent of students were offered tutoring and tutoring was most often 
provided by AP teachers (75% of students chose this option). Of the students who were offered 
tutoring (n = 635), 22% did not attend at all, 32% attended less than once a month, 29% attended 
about once a month, 12% attended about once a week, and 5% attended more than once a week. 
On a four-point scale (1= not at all useful, 2 = slightly useful, 3 = somewhat useful, and 4 = 
extremely useful) students on average rated the tutoring a 3.2 (SD = 0.71), which is an average 
rating of somewhat useful. 


Students were asked to indicate their level of agreement on a four-point scale (from 1 = 
strongly disagree to 4 = strongly agree) for a set of statements about their tutoring experiences. 
Students agreed most strongly on the content knowledge of their AP tutors (MV = 3.68) and the 
least about AP tutors working on their specific needs (M = 3.05), but in all cases the average 
level of agreement was above 3 (see Table 24). 
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Table 24 


Student Opinions on Tutoring 


Statement n M SD 
My AP tutors understand the content they are teaching. 486 3.68 0.55 
The tutoring improved my performance in the AP course. 487 3.21 0.68 
The AP tutors worked on my specific needs during the sessions. 484 3.05 0.83 
The tutoring sessions increase my confidence in my ability to successfully 487 3.10 0.77 
take the AP exams. 
It was easy to schedule AP tutoring sessions. 487 3.24 0.78 


5. Student Study Sessions 


Students were next asked a set of questions about the Saturday study sessions. Students 
were required to attend three student study sessions for each AP course in which they were 
enrolled, although we learned through some of the teacher interviews that many students had 
difficulty attending the Saturday sessions because of conflicting demands and scheduling issues. 
Looking only at the sample of students who completed the survey (n = 877) students took an 
average of two AP courses (the median was also two). Therefore, the required number of student 
study sessions, on average, was six (two courses x three study sessions each). Students reported 
attending an average of only 2.02 in-person study sessions (SD = 1.96), and an average of 0.30 
online study sessions (SD = 1.32). Thus, students were attending fewer than the required number 
of study sessions. Table 25 shows the average number of sessions students reported they 


attended by AP course subject. 
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Table 25 


Averages for In-person and Online Study Sessions Per AP Course 


In person Online 
Enrolled 

AP courses students Mean SD Mean SD 
English Language and Composition 472 1.89 1.48 0.15 1.20 
English Literature and Composition 185 1.33 1.46 0.03 0.22 
Calculus 205 1.88 3.20 0.13 0.53 
Computer Science 94 0.61 1.13 0.09 0.94 
Statistics 176 2.02 4.05 0.03 0.17 
Biology 185 1.09 1.49 0.12 0.75 
Chemistry 92 1.31 3.49 0.15 0.96 
Environmental Science 64 1.03 2.96 0.05 0.31 
Physics 270 1.60 1.56 0.10 0.46 


Students learned about the study sessions most frequently from their AP teachers (94% of 
the time) as well as other teachers and school counselors. Just over 70% of the students did not 
have the same instructor for the study sessions, and 75% of students indicated that students from 
other schools attended. Of the 822 students who responded, the average rating of usefulness of 


the student study sessions was 3.01 (SD = 0.80), or somewhat useful. 


6. Student Opinions About Study Sessions 


We asked students how much they agreed with statements related to the study sessions, for 
each AP course they were taking. Table 26 presents the average agreement of student 
respondents by subject area (English, math, and science). On average, students had the lowest 
level of agreement that the in-person study sessions were conveniently scheduled (means ranged 
from 2.88 to 2.98 on a four-point scale). The highest level of agreement varied by subject area 


(see Table 26). Average responses by AP subject area are included in Appendix G. 
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Table 26 


Student Responses to Study Session Statements 


Statement English overall Math overall Science overall 
The study sessions increased my confidence 3.14 (0.73) 3.16 (0.77) 3.02 (0.79) 
Improved my content understanding 3.16 (0.72) 3.18 (0.69) 3.05 (0.80) 
Conveniently scheduled 2.98 (0.87) 2.97 (0.86) 2.88 (0.89) 
Increased my confidence in my ability to take 
ik D cacene 3.09 (0.76) 3.09 (0.78) 2.97 (0.85) 
Improved my test-taking strategies 3.14 (0.77) 3.06 (0.81) 2.93 (0.87) 


Note. | = strongly disagree, 2 = disagree somewhat, 3 = agree somewhat, and 4 = strongly agree. 
7. AP Preparation 


Students were asked to indicate their level of agreement with a set of statements related to 
their preparation for the AP exams (see Table 27). Lowest levels of agreement were found for 
statements relating to outside school support for increasing content understanding (M = 2.89, 

SD = .88) and improving test-taking strategies (VM = 2.87, SD = 0.87). Students’ highest level of 
agreement was related to their AP teachers’ content understanding with an average agreement of 
3.66 (SD = 0.57). 


Table 27 


Student Opinions on Factors Relating to AP Preparation 


Statement n Mean SD 
My AP teachers understand the content they are teaching. 832 3.66 0.57 
The support the school provides outside of the classroom improved my study skills. 830 2.85 0.89 
Iam confident in my ability to successfully complete AP courses. 832 3.26 0.72 
Iam confident in my ability to successfully take AP exams. 832 3.10 0.77 


The support the school provides outside of the classroom increased my 


understanding of the course content. a2 aoe wee 
I was nervous about how hard the AP courses would be when I signed up for them. 829 3.18 0.88 
Iam confident in my ability to learn new STEM content. 831 3.22 0.72 
The support the school provides outside of the classroom improved my test taking 826 2.87 0.87 


strategies. 


Note. | = strongly disagree, 2 = disagree somewhat, 3 = agree somewhat, and 4 = strongly agree. 
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8. AP Exams 


Most students (n = 728) indicated they were planning on taking the AP exam for all their 
AP classes. We asked students who indicated they were not taking the exam the reason for their 
decision. Responses by general content area are shown in Table 28. In science, 70% of 
respondents said they were not ready to take the exam, compared to 42% of math test-takers and 
19% of English. Across all subject areas, fewer students cited college acceptance of AP scores to 
be a factor. Close to half of the respondents indicated they could not afford to take the exam, 


however fewer students indicated the cost of the AP exams was too much (see Table 28). 


Table 28 


Student Responses to Reasons for Not Taking AP Exams 


Science Math English 

Statement (n = 26) (n = 26) (n = 41) 

I don’t see any benefit in taking AP exams. 9 (31) 7 (29) 5 (24) 

Iam not ready to take my AP exam. 21 (70) 10 (42) 4 (19) 

I can’t afford to take the exam. 15 (50) 10 (42) 11 (52) 

Iam taking too many AP exams. 8 (27) 7 (28) 3 (14) 

My course load is too heavy to prepare for another test. 12 (40) 10 (42) 8 (38) 
The college I'm going to doesn't accept AP scores. 2 (7) 3 (12.5) 1 (5) 

The cost of all of my AP exams is too much. 9 (30) 5 (21) 3 (15) 


Note. The number in parentheses indicates the percentage (%) of students who chose each response option. 


9, Student Incentives and Rewards 


Students were asked which rewards were offered to them to participate in AP courses. 
Sixty-five percent of students were offered cash awards for performance on the AP exam and 
44% indicated they had their AP exam fee waived. Weighted grades (58%) and scholarship 
incentives (32%) were also cited as incentives. To earn the rewards two-thirds of the students 
indicated they had to pass the AP exam (with a score of 3 or higher), 34% had to enroll in an AP 
course, 48% had to complete and pass an AP course, and 40% indicated they had to take the AP 
exam. On a four-point scale students rated the importance of the incentives an average of 2.98 
(SD = 0.9) indicating the incentives (overall) were somewhat important. 


VI. Summary 


The impact findings for the first year of the independent evaluation of NMSI’s i3 scale-up 
grant indicated: 
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The percentage of students who took at least one AP exam was higher in schools 
implementing the CRP than in the control schools. However, this difference is not 
statistically significant. 


Also, the percentage of students scoring 3 or higher on at least one AP exam was lower 
in the treatment schools than in the control schools, however, this difference is not 
statistically significant either. 


These findings were confirmed by the small changes in the predicted probabilities of 
taking an AP course and scoring 3 or higher on an AP exam between the treatment and 
the control schools. 


A summary of findings from the teacher and administrator surveys and interviews follows: 


Student enrollment: Student financial incentives were seen to be the most effective at 
increasing student enrollment in AP courses, with open enrollment seen as the least 
effective. 


Student achievement: Teacher training was seen as the most effective component of the 
CRP related to student achievement and open enrollment the least effective. Teachers 
reported positive feedback on the training and indicated an increase in content 
knowledge. This suggests the CRP PD is meeting a need and helping prepare teachers to 
feel better equipped to teach AP courses. 


Study sessions: Student study sessions were viewed as an important and positive 
component of the program. The AP courses cover a lot of content and the study sessions 
were seen as helpful for providing students’ additional instructional time as well as 
teaching support and guidance for teachers. Many teachers mentioned in the interviews 
that the study sessions provided excellent resources for classroom use. Attendance at the 
study sessions was not as high as we would have expected, and one reason for this may 
be the scheduling issues. Students who responded to the survey indicated lower levels of 
agreement on the convenience of study session scheduling. Saturday scheduling posed 
significant problems for many students because of work and sports team commitments 
as well as other family-related obligations. 


Teacher incentives: While teachers consistently said incentives didn’t matter, they often 
clearly thought of them as (poor) compensation for the many additional hours of work 
involved in the program—particularly training and the student study sessions. Many 
teachers also reported that they would be teaching the AP courses regardless of the 
incentives—indeed some had been teaching the courses for several years. Teachers did, 
however, indicate that the incentives were nice to have and a welcome added bonus for 
their participation. 


Training: If the number of students enrolling in AP courses continues to increase, there 
may be a need for more training in differentiated instruction (for students of varying 
ability levels). 


School culture: Some schools in the sample indicated they had already been encouraging 
students to take AP courses and so the culture shift or improvement would not likely be 
as marked as in schools with no existing AP program. 
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Below we briefly summarize the key findings from the student survey measure: 


e Most students learned about AP and the CRP from their teachers or school counselors, 
and more than a third of students learned about the AP courses from other students. 


e Ninety-five percent of student respondents (n = 877) indicated they planned to attend 
some sort of postsecondary institution. 


e On a four-point scale (from 1= not at all useful to 4 = extremely useful) students on 
average rated the tutoring a 3.2 (SD = 0.71), which is an average rating of somewhat 
useful. 


e Students attended an average of 2.02 in-person study sessions, which was lower than the 
expected average of six. Of the 822 students who responded, the average rating of 
usefulness of the student study sessions was 3.01 (SD = 0.80) on a four-point scale, which 
is an average rating of somewhat useful. On average, students had the lowest level of 
agreement that the in-person study sessions were conveniently scheduled, which sheds 
additional light on the issue of poor attendance. 


e Students indicated their AP teachers had good content knowledge, and on average agreed 
with statements relating to their own confidence related to taking AP courses and exams. 


e Most students surveyed indicated they were planning on taking the AP exam and two 
thirds indicated they were offered cash incentives for performance on AP exams. 
On a four-point scale students rated the importance of the incentives an average of 2.98 


(SD = 0.9) indicating the incentives (overall) were somewhat important. 


A. Discussion 


The impact findings for the first year of the independent evaluation of NMSI’s i3 scale-up 
grant indicated a larger (although nonsignificant) percentage of students in the treatment schools 
taking at least one AP exam, compared to the control schools. In addition we found a higher 
percentage of students in the control schools achieved a 3 or higher on at least one AP exam 
(although again the difference was not significant). So ultimately, there were no differences 
between the two groups in terms of enrollment in AP and achievement on AP exams. As we 
were analyzing data from the first year of the CRP intervention it is perhaps not surprising we 
would not find a significant difference between the treatment and control schools. Students in the 
treatment schools may not have been exposed to a pipeline of high-quality STEM courses prior 
to participating in the CRP, and so they may not have had adequate time or opportunity to 
develop strong conceptual understanding of the material in their earlier coursework. Obtaining a 
qualifying score on an AP exam is challenging and perhaps even more so if the prerequisite 
knowledge is not present, or as strong as it could be. In addition, this study marked the first year 
teachers were implementing the CRP and attending the associated trainings. There is likely a 


learning curve for some teachers which may be reduced as they have more experience teaching 
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AP courses within the CRP. One of the goals of the CRP is to increase student enrollment in AP 
courses—particularly for students who may not typically see themselves as “AP students.” While 
we did see some higher levels of student enrollment in the treatment schools, these differences 


were not significant and so we cannot read too much into these findings. 


Data collected for the fidelity matrix indicated that not all elements of the CRP were 
implemented with high fidelity; however, overall results indicated that 23 schools (82.1%) 
achieved 80% or better implementation fidelity, for an average fidelity score of 89.3%. Seven 
schools achieved a perfect fidelity score of 100%. Looking at the different indicator groups 
(school supports, teacher supports, and student supports) we found that all school support 
measures across all schools were implemented with fidelity, with the caveat that complete data 
were only available for 16 schools (and incomplete data for 12 schools). So for the indicators on 
which we had data, all schools achieved above 80% fidelity in terms of school supports. Not all 
teachers were able to participate in all required training and so this teacher support component 
was not implemented with fidelity. Stipends and teacher awards were paid as expected as were 


student award payments. 


Teacher survey data indicated teacher training was seen as the most effective component of 
the CRP related to student achievement and open enrollment the least effective. Teachers 
reported positive feedback on the training and indicated an increase in content knowledge. This 
suggests the CRP PD is meeting a need and helping prepare teachers to feel better equipped to 
teach AP courses. Teacher incentives were chosen as the least important program component 
relating to increasing student performance by 16% of teachers and student incentives by 12% of 
teachers. Teachers did, however, view the student incentives as an important program component 
to encourage student enrollment in AP courses. Likewise, students rated the financial incentives 


on average as somewhat important in encouraging them to participate in AP courses. 


Student study sessions were viewed as important and positive components of the program. 
The AP courses cover a lot of content and the study sessions were seen as helpful for providing 
students’ additional instructional time as well as additional pedagogical support and guidance for 
teachers. Many teachers mentioned in the interviews that the study sessions provided excellent 
resources for classroom use. This makes ensuring adequate and accurate data on student 
attendance an important goal for Year 2, as well as discussing possible alternatives to attending 


these sessions at the weekends, in person. 


One of the goals of the CRP is to change school culture and to expand AP course access to 
traditionally underrepresented students. Teachers and administrators in some schools in the 


sample indicated they had already been encouraging students to take AP courses (prior to 
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implementation of the CRP) and so the culture shift or improvement would not likely be as 


marked as in schools with no prior AP program. 


Evaluation of the CRP from the teacher and administrator perspective provides both unique 
insight as well as valuable support for the fidelity of implementation data we are gathering as 
part of our evaluation study. Perspectives of those on the ground who are implementing the CRP 
helps determine which factors are most important in creating and sustaining an accessible and 
successful AP program. Survey and interview data from CRP teachers across the country 
provided valuable support for data collected for the implementation evaluation and helped 


determine how CRP participants perceive the importance and efficacy of key program elements. 


Findings presented in this report represent the first year of a multiyear evalution project 
designed to explore the impact of NMSI’s CRP as well as evaluate the fidelity of implementation 
and examine factors associated with successful implementation. We are currently in the second 
year of our evaluation study and during the 2017—2018 school year will be collecting data from 
the treatment schools from Year 1, as well as the Year 1 delayed treatment schools who in Year 


2 are receiving the treatment. 
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Table Al 


Appendix A: Connection to Logic Model 


Item Alignment to the Logic Model Component and Fidelity Matrix Construct 


Logic Model Component 


Question 


Survey Item 


Data Type/Analysis 


Teacher Components 
(direct) 


Training Do teachers perceive training to Q24: e, f, h, i Frequency (# of true 
be effective in terms of increasing ratings) 
their content knowledge and Q53:d,e Average ratings (3-point 
improving instructional scale) 
techniques? See Table 11 
Do teachers perceive attending Q36: g, h, i Average ratings (four- 
the study sessions as effective in point scale) 
increasing content 2 
knowledge/efficacy as AP ae 
teacher? 

Mentoring Do teachers perceive mentoring Q28: a, b,c, d Average ratings (four- 
as effective in increasing content point scale) 
knowledge/efficacy as AP 
teacher? 

Online Resources Did teachers have access to online Q38 Number of teachers who 
resources? selected option. 

How often did they access them? Q46 Frequency (4 choices) + 
open ended 

How did teachers use the online Q47 Frequency (7 choices) 

resources? 

Incentives/Awards How important were incentives in Q45 Average rating 
encouraging teaching AP? 

Overall Overall which elements of the Q50 Frequency of most 
program were most effective for Frequency of least 
increasing enrollment (school 
culture)? 

Overall which element of the Q51 Frequency of most 


program were most effective for 
increasing student performance in 
AP courses? (student success) 


Frequency of least 
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Logic Model Component 


Question 


Survey Item 


Data Type/Analysis 


Teacher Perceptions of 
Student Components 
(indirect) 


Focused Study Sessions Do teachers perceive attending Q36: d, f Average ratings (four- 
the study sessions as effective in Q37:a point scale) 
increasing student content 
knowledge and confidence? . 

Q53: a,b (3-point scale) 
-Incorporated 
into Table 13 

Equipment and supplies Do teachers have the materials / Q41 Percent “yes” 
equipment necessary to teach 
their courses effectively? : 

Q18:k Average ratings (four- 


See Table 10 


point scale) 


Exam Fee Subsidies 


N/A 


Student incentives (for 
exam performance) 


N/A (except for Q50 and Q51 but 
we already have those included 


above) 


Overall/culture 


Does the CRP effectively increase 
the participation of traditionally 


Q49 
Q52 


underrepresented populations? 


Is the CRP an effective way to 


Q53: ¢ 
See Table 11 


increase student enrollment in AP 


Percent “yes” 


Average ratings 


(3-point scale) 


courses? 

Q20: c, f, h Frequency 
Q18: e, j, m,n Average ratings (four- 
- Inserted in point scale) 
Table 16 
Q23: d,e 

Teacher Perceptions of 

School Components 

(indirect) 

Performance Analysis N/A 

Access to Academic & In what ways did the CRP Q48 Free response—we can 


Program Experts 


Program Manager help you to 


implement the CRP? 


collapse categories of 
comments. 
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Logic Model Component Question 


Survey Item 


Data Type/Analysis 


Does the school exhibit a culture 
of continuous improvement? 


Shared Goal Setting 


Do students value STEM 
learning? 


Do administrators value STEM 
learning? 


Q12 


QI5 
Q18: b, 1 


-Incorporated 
into Table 20 


Q23: b,c 


Q53: f 
See Table 11 


Q10: d,e 


Q18:d 
See Table 10 


Q53: g 
See Table 11 


Percent “yes” 


Average ratings (four- 
point scale) 


Average ratings 


(3-point scale) 


Average ratings (four- 
point scale) 


Average ratings (four- 
point scale) 


Average ratings 


(3-point scale) 
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Appendix B: Teacher Survey Measure 
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CRP Participating Teachers Survey 
Welcome to the AP Teacher Survey 


Dear AP Teacher, 


This survey is being conducted by the National Center on Research, Evaluation, Standards and 
Student Testing (CRESST) at UCLA as part of the implementation of the NMSI College 
Readiness Program (CRP) at your site. 


CRESST is collecting survey data from samples of AP teachers, AP coordinators, and students 
taking AP courses in partner schools and non-partner schools in several states. The survey should 
take approximately 30 minutes to complete. 


The purpose of the surveys is to understand the perspectives of these individuals on the 
implementation and impact of AP courses in their schools and, where appropriate, on the 
implementation of the College Readiness Program. It is not intended to evaluate your individual 
capabilities or performance. Responses will be analyzed to provide information about how AP 
courses are administered across the country. 


We thank you for taking part in our survey. Your participation and feedback is very important. 
As a thank you for participating, upon completion of the survey you will receive a $50 gift card. 


Please direct any questions about the project to Julia Phelan Ph.D (Julia.phelan @ucla.edu, 310- 
206-4998). 


Thank you! 


Teacher Background 
Please consider your own personal experience when answering the following questions. 


1. In which school do you currently teach? 


2. Please select the one title that best describes your current position. 

O Classroom teacher (1-2 years teaching experience) 

O Classroom teacher (3 or more years teaching experience) 

O Out-of-classroom position such as instructional coach or district personnel 
O Special Education teacher 

O Other (please specify) 


3. At which grade levels do you teach? (Please check all that apply.) 
O19 

O10 

ime 

O12 


CL) Other (please specify) 
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4. To the nearest year, how long have you: 
been a teacher? 

been a teacher in your current school? 
been a teacher in your current subject? 
been a teacher in your current district? 
been a teacher of AP courses? 


5. In which content area(s) do you teach? (Please check all that apply.) 
C1 Physical Science 

C1 Earth Science 

C1 Life science 

CL] Chemistry 

CL] Computer Science 

XO) Calculus 

LO Algebra 

C1 Trigonometry 

C1 Integrated Mathematics 
C1 Statistics 

CL English Language Arts 
CL) Other (please specify) 


6. Specifically in your AP classes, how many students do you have in each grade? 


Grade 9 

Grade 10 
Grade 11 
Grade 12 


7. Which type of teaching credentials(s) do you hold? (Please check all that apply.) 
CO) Clear Credential 

XO) Preliminary Credential 

L] Emergency Credential 

CL Multiple Subjects 

C1 Single Subject 

C1 National Board Certification 

C1 Currently in a program to obtain my teaching credential 

CL) None 

C1 Other (please specify) 
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8. In the past three years, have you participated in any of the following general professional 
development training opportunities? (Please check all that apply.) 

LX) Subject-area content 

L] Pedagogy / instruction 

XL) Subject-area curriculum 

CL) Data analysis and using data to modify instruction 

C1 Improving students’ critical thinking or problem solving skills 

CL] None 

C1 Other (please specify) 


9. How much do you agree or disagree Strongly Agree Disagree Strongly 
with the following statements? Agree Somewhat Somewhat Disagree 
I have a strong sense of belonging to the O O O O 


community of educators. 


I have a good understanding of the 
concepts I need to teach in my field. 


I learn new ideas in my field quickly. 


in learning their AP course content. 


O O O Q 
O O O O 
My students are usually pretty interested O O O O 
O O O O 


My students put in the effort it takes to 
learn in their AP course. 


10. In which content area(s) are you teaching AP courses this year? (Please check all that apply.) 
CL) Physics 

C1 Earth Science 

XO Life science 

CL) Chemistry 

CL) Computer Science 

XO) Calculus 

CX) Statistics 

C1 English Language and Composition 
C1 English Literature and Composition 
C1 Other (please specify) 


11. Besides collaboration during Vertical Team meetings, do you collaborate with other AP 
math, science, and English teachers as part of your AP teaching (e.g., in a professional learning 
community [PLC] or in team meetings)? [“No” skips to item #15.] 

O Yes 

O No 
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Teacher Collaboration 

12. Who typically leads these meetings? 
O AP coordinator 

O Administrator(s) 

O AP teacher(s) 

O Content Specialists 

O Leadership rotates among participants 
O Other (please specify) 


13. What topics are typically addressed at these meetings? (Choose all that apply.) 
C1 Subject specific content 

C1 Pacing and/or sequencing information 

C1 AP test preparation 

CL] Data analysis and use to modify instruction 

CL] Development of common assessments 

C1 Other (please specify) 


14. How would you describe the usefulness of the meetings? 
CL Extremely useful 

01 Useful 

CL) Somewhat useful 

C1 Not at all useful 
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Teaching Practices and Beliefs 
15. Please indicate whether you Strongly Agree Disagree Strongly 
agree/disagree with the following Agree Somewhat Somewhat Disagree 
strategies as effective ways of 
strengthening your students’ content 
understanding. 


Teacher as “facilitator” or “coach”. O O O O 
Peer-to-peer problem solving. O O O O 
Teacher asking students guiding O O O O 
questions. 

Teacher providing time for students to O O O O 
persevere through solving complex 

problems. 

Teacher demonstrating the correct way to O O O O 
solve a problem. 

Teacher facilitating students’ own inquiry. 2) O O O 
Instruction built around problems with O O O O 


clear, correct answers, and around ideas 
that most students can grasp quickly. 

How much students learn depends on how O O O O 
much background knowledge they have — 
that is why teaching facts is so necessary. 


Thinking and reasoning processes are O O O O 
more important than specific curriculum 

content. 

Teachers shouldn’t let students develop O O O O 


answers that may be incorrect when they 
can just explain the answers more directly. 
Students should be allowed to think of O O O O 
solutions to practical problems themselves 
before the teacher shows them how they 
are solved. 


16. What supports (if any) are offered to you specifically as an AP math, science, or English 
teacher? (Choose all that apply.) 

O Additional planning time 

O Increased access to texts and other print resources 

O Increased access to Web-based resources 

O Increased access to technology 

O Additional professional development (PD) opportunities 

O Increased access to collaboration with other AP teachers 

O None 

O Other (please specify) 
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17. How much do you agree or disagree 
with the following statements? 


Strongly 
Agree 


Agree 
Somewhat 


Disagree 
Somewhat 


Strongly 
Disagree 


The students in my school are well- 
prepared to take AP courses. 


O 


O 


O 


O 


My school promotes a culture of 
continuous improvement. 


My school offers too many AP courses. 


My school leadership values STEM 
learning. 


Many students who may do well in AP 
courses do not take them. 


My school offers too few AP courses. 


Students in AP courses in my school are 
well-prepared for the exam. 


Students in AP courses in my school 
believe that they are well-prepared for the 
exam. 


O| OO; O| OBO; O 


oO} OO; O| OBO; O 


Oo} OO; O| OBO; O 


QO}. Orolh “Or <Or@) <O 


My school sets clear goals for AP 
instruction and exam performance. 


My school sets goals for the number of 
students participating in AP courses and 
taking AP exams. 


My school provides high quality 
instructional materials for teaching AP 
courses. 


My school provides quality instruction on 
strategies and techniques for teaching AP 
courses. 


My school encourages all students to 
enroll in AP courses. 


My school considers all students — 
regardless of gender, ethnicity, or 
socioeconomic status — as capable of 
achieving at high levels. 


18. Did your school offer AP courses in the 2015-2016 school year? [“No” skips to #20. ] 


O Yes 
O No 
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AP Course History 

19. Please select the most important reasons why qualified students in your school have not taken 
AP courses in the past? (Choose all that apply.) 

L] The students don't know about AP courses. 

CL My school has not previously offered enough courses. 

CL] My school does not allow all interested students to enroll. 

CL) Students can't afford to take the year-end exam. 

C1 Students do not meet the requirements to enroll in AP classes. 

C1 Student's don't realize that they are qualified. 

C1 The courses have reputations as being too difficult. 

C1 School administration and teachers do not perform enough outreach to promote AP courses. 
CL) Other (please specify) 


Required Goal Setting Meeting 

20. Did your school hold a shared goal setting meeting in the fall to discuss targets for student 
performance and enrollment in AP courses? [“No” skips to #23.] 

O Yes 

O No 


Goal Setting Meeting Attendance 

21. Did you attend the shared goal setting meeting? [“No” skips to #23.] 
O Yes 

O No 


Evaluation of Goal Setting Meeting 


22. Please indicate how strongly you Strongly Agree Disagree Strongly 
agree or disagree with the following Agree Somewhat Somewhat Disagree 
statements about the shared goal setting 
meeting. 

The meeting was effective 


O}O 
O}O 
O}O 
O}O 


The meeting established measurable and 
attainable goals for class enrollment 
The meeting established measurable and O O O O 
attainable goals for student exam 
performance 


The meeting established goals for O O O O 
recruitment of high-need and traditionally 
underrepresented students into AP courses 
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Teacher Training Activities 
23. Please indicate which of the following Four-Day AP Two-Day One-Day 
statements are true for each of the CRP teacher Summer Fall Spring 
training activities listed. (Select all that apply.) Institute (APSI) | Workshop Training 
I attended the training 
I attended all days of the training 
The facilitators were knowledgeable and well- 
prepared 
The location suited the needs of the training 
activities 
The training activities improved my content 
knowledge 
At the end of the training, I felt more qualified to 
be an effective AP instructor 
The agenda and goals of the training activities 
were clear 
The training activities were effective 
The training helped me differentiate instruction 
for students at differing ability levels 


OO, Ua oo Oo ooo 
Oo, a oF oOo ooa 
Oo, UU oo oa ooo 


24. As part of the CRP, was Teacher Coaching or Mentoring made available to you? [“No” skips 
to #28.] 

O Yes 

O No 


Teacher Mentoring Frequency 

25. How often do you meet or communicate (virtually or in-person) with your assigned mentor? 
[“Not at all” skips to #28.] 

O Daily 

O Weekly 

O Monthly 

O Not at all 

O Other (please specify) 


Teacher Coaching or Mentoring 

26. What type of support did your mentor (CRP content specialist) provide? 
CL) Modeling/demonstrating lessons 

C1 Information on pacing and sequencing of lessons 

C1 Help designing lessons or using course texts 

CL) Advice on differentiating instruction for students at differing ability levels 
LC) Direct instruction or tutoring with CRP students 

C1 Other (please specify) 
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27. Please indicate how strongly you Strongly Agree Disagree Strongly 
agree or disagree with the following Agree Somewhat Somewhat Disagree 
statements about the CRP mentoring. 

The mentor was knowledgeable and well- O O O O 
prepared 

The mentoring improved my content O O O O 
knowledge 

The mentoring honed my instructional O O O O 
skills and techniques 

Because of the mentoring I am a more O O O O 
effective and qualified AP instructor 

The mentoring sessions were conveniently O O oO O 
scheduled 

The mentor’s guidance on pacing was O O O O 
helpful 

I would like to have met more frequently O O O O 
with my mentor 

My mentor did not have much time to O O O O 
meet with me 


Tutoring Session Availability 

28. As part of the CRP, does your school offer tutoring sessions (e.g., one-on-one or small group 
academic help) outside of AP class time (either within or outside of the school day)? [“No” skips 
to #31.] 

O Yes 

O No 


Participation in Tutoring 

29. How are students identified for tutoring? (Choose all that apply.) 
XL) Scores on classroom assessments 

L) Classroom performance 

C1 Scores on standardized or other large-scale tests 

CL Other academic criterion 

CL] Other data, such as behavioral and/or attendance 

LC) Referrals from other teachers or school staff 

LC Self-identified (i.e. students volunteer to attend) 

C1 Other (please specify) 


30. Approximately, how many of your AP students typically attend tutoring sessions? 
O 75% - 100% 

O 50% - 74% 

O 25% - 49% 

O < 25% 

O None 
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Instructional Supports 

31. In general, to what extent do you encourage students to attend student study sessions? 
O To a great extent 

O To a moderate extent 

O To a small extent 

O Not at all 


32. How many student study sessions did you attend this year? 
O 5 or more 

O3o0r4 

O lor2 

O None 


33. Students are required to attend at least three student study sessions outside of the regular 
school day. Approximately what percentage of your AP students satisfied this requirement? 
O 75% - 100% 

O 50% - 74% 

O 25% - 49% 

O < 25% 

O None 


34. Please indicate the type of study 75-100% 50-74% 25-49% <25% 
sessions attended by your students. 


Saturday, in-person study sessions O O O O 
Online study sessions O O O O 
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35. Please indicate the extent to which 
you agree or disagree with each of the 
following statements about the student 
study sessions. 


Strongly 
Agree 


Agree 
Somewhat 


Disagree 
Somewhat 


Strongly 
Disagree 


My input was considered when 
determining the study session topics 


O 


©) 


S) 


O 


Students were active participants (e.g., 
answering and asking questions, focused 
on tasks assigned, etc.) 


O 


O 


The study sessions were conveniently 
scheduled to accommodate student 
schedules 


The study sessions improved students’ 
content knowledge 


The study sessions were led by AP experts 
who taught NMSI-created lessons 


The study sessions helped to increase 
student confidence 

The study sessions highlighted the 
instructional needs of the students for me 
to continue addressing in class 

I learned a great deal from watching the 
expert teachers during the student study 
sessions 

I was able to take the strategies I saw 
employed during the study sessions back 
to my own classrooms to improve student 
achievement 


36. Please answer the following 
questions about how useful the student 
study sessions have been. 


Extremely 
Useful 


Useful 


Somewhat 
Useful 


Not at All 
Useful 


In general, how would you describe the 
usefulness of the student study sessions 
for students? 


O 


S, 


O 


O 


In general, how useful are the student 
study sessions for you? 


How useful was it to include an online 
student study session as well as in-person 
sessions? 
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37. In addition to regular classroom materials, what instructional resources are offered for use 
specifically in your AP math, science, or English class? (Choose all that apply.) 

CL) Supplementary texts to the core curricular materials 

XO) Relevant articles and/or monographs 

C1 College Board AP curricular materials 

C1 Access to online resources 

CL Technology 

C1 Access to field experiences for AP students (e.g., meeting with local government leaders, 
conducting water quality tests at local stream) 

C1 Do not know 

CL) None 

C1 Other (please specify) 


38. What assessments does CRP offer specifically for your AP math, science, or English class? 
(Choose all that apply.) 

CL) Frequent classroom short assessments to monitor student progress 
CL) AP-level multiple-choice-item guides 

LC) AP-level essay prompts 

LI Practice AP tests 

C1 Scoring standards and/or rubrics 

C1 Unit tests 

XO) Do not know 

CL] None 

C1 Other (please specify) 


39. What equipment does CRP offer for use specifically in your AP math, science, or English 
class? (Choose all that apply.) 

CL) Smartboard or other presentation device 

CO) Lab or classroom supplies 

C1 Student review materials 

CL) Additional computer hardware, such as printers and scanners 

XO Do not know 

CL) None 

C1 Other (please specify) 


40. Do you have the materials/equipment you need to teach your course effectively? 
O Yes 
O No 


Teacher Incentives Availability 

41. Were incentives (financial or otherwise) offered to you as an AP math, science, or English 
teacher? [‘‘No” skips to #45.] 

O Yes 

O No 
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Teacher Incentives in Practice 

42. What incentives were offered to you as an AP math, science, or English teacher? (Choose all 
that apply.) 

CL] Financial incentives 

CL) Additional PD opportunities 

C1 Recertification points 

CL) Leadership position on school PLC or other team 

C1 Other (please specify) 


43. For what milestones or achievements were incentives offered? (Choose all that apply.) 
C1 Total number of students enrolling in an AP course 

C1 Total number of students completing and passing an AP course (score of 3 or above) 

C1 Total number of students taking AP exams 

C1 Student participation in additional study work (e.g., tutoring and weekend sessions) 

C1 Attendance at certain training sessions or other similar events 

C1 Serving in a leadership role on PLCs or other school committees 

CL Do not know 

CL) Other (please specify) 


44. How important were these incentives in encouraging your teaching of AP math, science, or 
English classes? (Choose one.) 

O Extremely important 

O Important 

O Somewhat important 

O Not at all important 


Accessing Online Resources 

45. As part of the CRP, online instructional materials are available to teachers via 
NMSlITeachers.org. How often do you access the CRP online instructional materials? [“No” 
skips to #47.] 

O Daily 

O Weekly 

O Monthly 

O Not at all 

O Other (please specify) 


Using Online Resources 

46. How did you use instructional materials on NMSITeachers.org? (Select all that apply.) 
L) To assess student progress 

C1 To familiarize students with the types of questions on the AP exam 

C1 To deepen the instruction in particular content areas 

L To teach test taking strategies 

C1 To prepare for upcoming student study sessions 

C1 To conduct a practice AP exam 

L) To review the materials covered in the teacher training sessions 
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47. In what ways did the CRP Program Manager help you to implement the CRP? [Free 
response] 


48. Do you feel that the CRP is an effective way to increase student enrollment in AP courses? 
O Yes 

O No 

Additional comments: 


49. Which elements of the CRP were Open Teacher Student Teacher 
most/least effective at increasing student | Enrollment | Training Financial Incentives 
enrollment in AP courses? Awards 
Most effective O O O © 
Least effective O O O O 
50. Which elements of Open Teacher | Student Teacher Student Online 
the CRP were most/least | Enrollment | Training | Financial | Incentives Study Instructional 
effective at increasing Awards Sessions Resources 
student performance in 
AP courses and on AP 
exams? 
Most effective O O O O O O 
Least effective O O O O O O 

51. Overall, has opening up enrollment to all students had a positive impact on the AP program? 

O Yes 

O No 
52. Indicate whether the CRP contributed to Major Slight No 
improving the following: Improvement | Improvement | Improvement 


Students' content knowledge 

Students' experience with STEM AP courses 
Recruitment of high-need and traditionally 
underrepresented students into AP courses 
Teachers' content knowledge 

Teachers’ instructional skills, techniques and 
strategies 

School culture of continuous improvement 
School leadership valuing STEM learning 


O;O| O;O} OOO 
O;O| O;O} OOO 
O;}O| O;O} O;O/O 


53. If you feel there were factors which significantly helped or impeded your ability to 
successfully implement the College Readiness Program, please describe them below. 


The survey is complete! Thank you for participating--we appreciate your time and your expert 
feedback. 
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Appendix C: Administrator Survey Measure 
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CRP Administrator Survey 
Welcome to the CRP/NMSI Administrator Survey 


Dear School Administrator, 


This survey is being conducted by the National Center on Research, Evaluation, Standards and 
Student Testing (CRESST) at UCLA as part of the implementation of the NMSI College 
Readiness Program (CRP) at your site. 


CRESST is collecting survey data from samples of AP teachers, AP coordinators, and 
students taking AP courses in partner schools and non-partner schools in several states. The 
survey should take 20-25 minutes to complete. 


The purpose of the surveys is to understand the perspectives of these individuals on the 
implementation and impact of AP courses in their schools and, where appropriate, on the 
implementation of the College Readiness Program. It is not intended to evaluate your 
individual capabilities or performance. Responses will be analyzed to provide information 
about how AP courses are administered across the country. 


We thank you for taking part in our survey. Your participation and feedback is very 
important. 


Please direct any questions about the project to Julia Phelan, Ph.D. Gulia.phelan @ucla.edu, 
310-206-4998). 


Thank you! 


AP in Your School 
1. For which school are you the designated CRP Administrator? 


2. How many years have you been coordinating Advanced Placement (AP) courses at your 
current school? (If this is your first year, please select “One year.”) (Choose one.) 

CL One year 

LC) Two to four years 

L] Five years or more 

C1 Not applicable (please specify) 
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3. What supports (if any) are offered to AP math, science, or English teachers? (Choose all 
that apply.) 

O Additional planning time 

O Increased access to texts and other print resources 

O Increased access to Web-based resources 

O Increased access to technology 

O Additional professional development (PD) opportunities 

O Increased access to collaboration with other AP teachers 

O None 

O Other (please specify) 


4, Did your school hold a shared goal setting meeting in the fall to discuss targets for student 
performance and enrollment in AP courses? ["No" skips to #7.] 

O Yes 

O No 


Goal Setting Meeting Attendence 

5. Did you attend the shared goal setting meeting? ["No" skips to #7.] 
O Yes 

O No 


Evaluation of Goal Setting Meeting 


6. Please indicate how strongly you agree or | Strongly Agree Disagree | Strongly 
disagree with the following statements Agree Somewhat | Somewhat | Disagree 
about the shared goal setting meeting. 

The meeting was effective O O O O 
The meeting established measurable and oO O O O 
attainable goals for class enrollment 

The meeting established measurable and O O O '®) 
attainable goals for student exam 

performance 

The meeting established goals for O O O © 


recruitment of high-need and traditionally 
underrepresented students into AP courses. 
The meeting established goals for providing O O O O 
equitable access to AP coursework for all 
interested students. 
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Program Coordination 

This section asks about how you coordinate the administration of the College Readiness 
Program. Please keep in mind that these items ask specifically about activities related to CRP, 
and not similar activities you may conduct as part of your work with other programs. 


7. What specific materials and resources have you provided or obtained for the CRP this year? 
(Choose all that apply.) 

O Core and/or supplemental texts 

O Other literature (either digital or print resources) 

O Technology 

O Lab equipment 

O Other (please specify) 


8. How do you support the coordination of the CRP classes? (Choose all that apply.) 
O Designing the AP course schedule 

O Assigning teachers to AP courses 

O Enrolling students in AP classes 

O Leading and/or assisting with tutoring or other direct instruction 

O Leading and/or assisting with AP exam preparation 

O Other (please specify) 


9. How do you support the administration of AP exams? (Choose all that apply.) 
O Ordering and/or distributing exams 

O Ensuring exams are administered according to College Board requirements 

O Proctoring test administration 

O Tracking and ensuring waiving of exam fees (when appropriate) 

O Other (please specify) 


10. To what extent do you support and To aGreat | To a Moderate | To a Small | Not at 
advocate for the CRP in the following Extent Extent Extent All 
activities? 

Participating in school AP fairs O O O O 
Meeting directly with students and oO O O O 
parents 

Conducting promotional activities with O O O O 
the state, district, or other external 

entities 


fie) 


Program Oversight 

This section asks about your role working with the CRP teachers and students. Again, these 
items ask specifically about this work as it relates to CRP, not other activities around which you 
may interact with these individuals. 


11. To what extent do you provide To a Great Toa Toa Small | Not at All 
program oversight in the following Extent Moderate Extent 

activities to ensure teachers are adhering Extent 

to the College Board content-specific 

curriculum? 


Analysis of student data (including AP 
exam scores) 
Classroom observations 


Teacher-developed reports 
Review of teacher portfolios 


O;O;O} O 
O;O;O} O 
O/O;O} O 
O;O;O} O 


12. How have your activities related to AP coordination changed since the implementation of 
the CRP? (Choose all that apply. Please specify.) 

Provided new or different materials 

Performed new or different tasks to coordinate AP classes 

Performed new or different tasks to coordinate AP exams 

Performed new or different tasks to support and promote the CRP 

Utilized new/different strategies to provide program oversight of adherence to the College 
Board curriculum 

Utilized new/different strategies to provide program oversight of support to students 


Other 
No changes (Please enter "no" in the text box.) __ 


13. To what extent do you provide program | To a Great Toa ToaSmall | Not at All 
oversight in the following activities to Extent Moderate Extent 
ensure students are receiving the support Extent 


they need to succeed in CRP classes? 


Analysis of student data (including AP O O O O 
exam scores) 

Classroom observations O O O O 
Conferences with CRP teachers 6) O O O 
Conferences with parents O O O O 
Conferences with students O O O O 


Instructional Supports 

14. As part of the CRP, were students offered exam fee subsidies or other financial awards? 
O Yes 

O No 
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15. In addition to regular classroom materials, what instructional resources are offered to 
teachers for use specifically in their AP math, science, or English class? (Choose all that 
apply.) 

CL) Supplementary texts to the core curricular materials 

C1 Relevant articles and/or monographs 

XL College Board AP curricular materials 

C1 Access to online resources 

CL) Technology 

L Access to field experiences for AP students (e.g., meeting with local government leaders, 
conducting water quality tests at local stream) 

LO Do not know 

LI None 

CL] Other (please specify) 


16. Do teachers have the materials/equipment they need to teach their courses effectively? 
O Yes 
O No 


17. Were you offered financial incentives as the CRP administrator? ["No" skips to #22. ] 
O Yes 
O No 


Administrator Incentives in Practice 

18. For what milestones or achievements are incentives offered? (Choose all that apply.) 
LO Total number of students enrolling in an AP course 

CL) Total number of students completing and passing an AP course 

C1 Total number of students taking AP exams 

C1 Student participation in additional study work (e.g., tutoring and weekend sessions) 
CL Attendance at certain training sessions or other similar events 

C1 Serving in a leadership role on PLCs or other school committees 

LO Do not know 

XO) Other (please specify) 


19. How important are these incentives in encouraging your participation in the CRP? 
(Choose one.) 

O Extremely important 

O Important 

O Somewhat important 

O Not at all important 


2015-2016 AP Courses 

20. Did your school offer AP classes in the 2015-2016 school year? ["No" skips to #22.] 
O Yes 

O No 
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AP Course History 

21. Please select the most important reasons why qualified students in your school have not 
taken AP courses in the past? (Choose all that apply.) 

O The students don't know about AP courses. 

O Our school has not previously offered enough courses. 

O Our school does not allow all interested students to enroll. 

O Students can't afford to take the year-end exam. 

O Students do not meet the requirements to enroll in AP classes. 

O Student's don't realize that they are qualified. 

O The courses have reputations as being too difficult. 

O We have not performed enough outreach to promote AP courses. 
O Other (please specify) 


CRP Administration 
22. Did CRP Program Managers provide sufficient support Yes No 
coordinating each of the following? 
Delivery of student study session reminders and materials O O 
Assistance in teacher sign-ups for summer trainings, two-day O O 


trainings, student study sessions 
Assignment of mentor teachers 


Guidance and direction concerning NMSI provided curricular 
support materials 


O}O 
O}O 


23. Do you feel that the CRP is an effective way to increase enrollment in AP courses? 
O Yes 

O No 

Additional comments 


24. Which elements of the CRP were Open Teacher Student Teacher 
most/least effective at increasing Enrollment Training Financial Incentives 
student enrollment in AP courses? Awards 

Most effective O O O O 
Least effective O O O O 


25. What elements of Open Teacher | Student Teacher Student Online 
the CRP were Enrollment | Training | Financial | Incentives Study | Instructional 
most/least effective Awards Sessions | Resources 
at improving student 

performance in AP 

courses and on AP 

exams? 

Most effective O O O O O O 
Least effective O O O O O O 


26. Overall, has opening up enrollment to all students had a positive impact on the AP program? 


O Yes 
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O No 


27. Indicate whether the CRP contributed to 
improving the following: 


Major 
Improvement 


Slight 
Improvement 


No 
Improvement 


Students’ content knowledge 


Students experience with STEM AP courses 


Recruitment of high-need and traditionally 
underrepresented students into AP courses 


Teachers’ content knowledge 


Teachers’ instructional skills, techniques and 
strategies 


School culture of continuous improvement 


School leadership valuing STEM learning 


O;/O| O;O; O;O;O 


O;/O| O;O; OOO 


O;/O| O;O; OOO 


28. If you feel there were factors which significantly helped or impeded your ability to 
successfully implement the College Readiness Program, please describe them below. 


The survey is complete! Thank you for participating - we appreciate your time and your 


expert feedback. 
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Appendix D: Student Survey Measure 
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CRP Study: AP Student Survey 
Directions: Please write-in your response or circle the letter that best fits your answer. 
1. What school do you attend? 


2. How many AP math, science, and English courses have you taken in high school including the 
courses in which you are currently enrolled? 


3. Please indicate which AP course(s) are you currently taking? 
Course Name 


English Language and Composition 


English Literature and Composition 


Calculus 


Computer Science 


Statistics 


Biology 


Chemistry 


Environmental Science 


Physics 


4. How did you learn about the AP courses available at your school? (Choose all that apply.) 
L] My AP teacher(s) 

CL] Other teacher(s) in the school 

CL A school counselor 

CL] Other student(s) 

XO) School signs, emails or fliers 

LX People outside of the school (e.g., private tutor, teachers from other schools) 

Please specify (roles, not names): 

CL] Other 

Please specify (roles, not names): 
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5. How did you learn about the College Readiness Program? (Choose all that apply.) 
L] My AP teacher(s) 

CL) Other teacher(s) in the school 

QO) A school counselor 

CL] Other student(s) 

XO) School signs, emails or fliers 

L) People outside of the school (e.g., private tutor, teachers from other schools) 
Please specify (roles, not names): 

CL) Other 

Please specify (roles, not names): 


6. What is the highest level of education you plan to complete? (Choose one.) 
O High school 

O Career/technical school 

O 2 year community/junior college 

O 4 year college or university 

O Graduate school - MD/PhD/MA/MBA 


Student Supports 


This section asks you about any supports available to you as an AP student to help you successfully 


complete AP math, science, and English courses for this school year. 
Tutoring 


We will start by asking about any tutoring supports (e.g., one-on-one or small group academic help) you 
receive. These tutoring supports do not refer to online and weekend Study Sessions. We will ask you 


about Study Sessions later in the survey. 


7. Are tutoring sessions, either during or outside of the school day, offered for any of your AP math, 


science, or English classes? (Choose one.) 
O Yes 
O No (Skip to item 14.) 
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8. Who typically provides these tutoring sessions? (Choose all that apply.) 

L] My AP teacher(s) 

L) Other teacher(s) in the school 

L] Other student(s) 

C1 People outside of the school (e.g., private tutor, teachers from other schools) 
Please specify (roles, not names): 

CO) Other 

Please specify (roles, not names): 


9. How did you learn about the tutoring sessions? (Choose all that apply.) 

L] My AP teacher(s) 

CL) Other teacher(s) in the school 

QO) A school counselor 

L] Other student(s) 

XO) School signs, emails or fliers 

C1 People outside of the school (e.g., private tutor, teachers from other schools) 
Please specify (roles, not names): 

CL) Other 

Please specify (roles, not names): 


10. How often do you attend any tutoring sessions for your AP math, science, or English courses? 
(Choose one.) 

O More than once each week 

O About once each week 

O About once each month 

O Less than once each month 

O Never (Skip to item 14.) 


11. Did you have the same tutor for each session in a subject? (Choose one.) 


O Yes 
O No 
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12. In general, how would you describe the usefulness of these tutoring sessions? (Choose one.) 


O Extremely useful 
O Somewhat useful 
O Slightly useful 
O Not at all useful 


13. Please indicate how strongly you agree 
or disagree with each of the following 
statements: 


My AP tutors understand the content they 
are teaching 


The tutoring improved my performance in 
the AP course 


The AP tutors worked on my specific needs 
during the sessions 


The tutoring sessions increase my 
confidence in my ability to successfully 
take the AP exams 


It was easy to schedule AP tutoring sessions 


Strongly Agree Disagree Strongly 
Agree Somewhat Somewhat Disagree 
O O O O 
O O O O 
O O O 0 
O O O O 
O O O O 
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AP Study Sessions 
Now, we will ask you about the supports offered in Study Sessions. 


14. Please write in the number of Study 
Sessions you completed online and attended in 


person for each of the following AP classes In-person Study Online Study 
you are taking. Sessions Attended Sessions 


English Language and Composition 


English Literature and Composition 


Calculus 


Computer Science 


Statistics 


Biology 


Chemistry 


Environmental Science 


Physics 


15. How did you learn about the Study Sessions? (Choose all that apply.) 

L] My AP teacher(s) 

CL) Other teacher(s) in the school 

QO) A school counselor 

L] Other student(s) 

C1 School signs, emails or fliers 

C1 People outside of the school (e.g., private tutor, teachers from other schools) 
CL) Other 

Please specify (roles, not names): 


16. Did you attend any in-person Study Sessions? (Choose one.) 


O Yes 
O No (Skip to item 19.) 
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17. Did you have the same instructor at each in-person Study Session for a subject? (Choose one.) 
O Yes 
O No 


18. In general, how would you describe the usefulness of Study Sessions? (Choose one.) 
O Extremely useful 

O Somewhat useful 

O Slightly useful 

O Not at all useful 


19. Please indicate how strongly you agree or disagree with each of the following statements about the 
science AP class you are taking (if any): 


Name of science AP course: Strongly Agree Disagree Strongly 
Agree Somewhat Somewhat Disagree 


The Study Sessions increased my e) O O O 
confidence in my ability to 
successfully complete AP courses 


The Study Sessions improved my O O O O 
understanding of the course content 

The in-person Study Sessions were QO O O O 
conveniently scheduled 

The Study Sessions increased my Q O O O 


confidence in my ability to 
successfully take the AP exams 


The Study Sessions improved my O O O O 
test taking strategies 
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20. Please indicate how strongly you agree or disagree with each of the following statements about the 
math AP class you are taking (if any): 


Name of math AP course: Strongly Agree Disagree Strongly 
Agree Somewhat Somewhat Disagree 


The Study Sessions increased my O O O O 
confidence in my ability to 
successfully complete AP courses 


The Study Sessions improved my O O O O 
understanding of the course content 

The in-person Study Sessions were O O O O 
conveniently scheduled 

The Study Sessions increased my O O O © 


confidence in my ability to 
successfully take the AP exams 


The Study Sessions improved my O O O O 
test taking strategies 


21. Please indicate how strongly you agree or disagree with each of the following statements about the 
English AP class you are taking (if any): 


Name of English AP course: Strongly Agree Disagree Strongly 
Agree Somewhat Somewhat Disagree 


The Study Sessions increased my O O O O 
confidence in my ability to 
successfully complete AP courses 


The Study Sessions improved my O O O O 
understanding of the course content 

The in-person Study Sessions were O O S) O 
conveniently scheduled 

The Study Sessions increased my O O O O 


confidence in my ability to 
successfully take the AP exams 


The Study Sessions improved my O O O O 
test taking strategies 
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Student Preparation 
In this section, we want to know how you feel about your own preparation for the AP exams. 
22. Please indicate how strongly you agree or disagree with each of the following statements: 


My AP teachers understand the 
content they are teaching 


The support the school provides 
outside of the classroom improved 
my study skills 


I am confident in my ability to 
successfully complete AP courses 


I am confident in my ability to 
successfully take AP exams 


The support the school provides 
outside of the classroom increased 
my understanding of the course 
content 


I was nervous about how hard the 
AP courses would be when I signed 
up for them 


I am confident in my ability to learn 
new STEM content 


The support the school provides 
outside of the classroom improved 
my test taking strategies 


Strongly Agree Disagree Strongly 
Agree Somewhat Somewhat Disagree 

O O O O 

O O O O 

O O O O 

O O O O 

6) O O O 

O O O © 

O O O O 

O O O O 


23. Are you planning to take the AP exams for all of your math, science and English AP courses? 
O Yes (Skip to item 28.) 


O No 
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24. If you are NOT planning to take your science AP exam, please indicate whether the following 
statements are true or false. 


Name of science AP course: 


I don’t see any benefit in taking AP exams. 


I am not ready to take my science AP exam. 


I can afford to take the science exam. 


I am taking too many AP exams. 


My course load is too heavy to prepare for another test. 


The college I’m going to doesn’t accept AP scores. 


The cost of all of my AP exams is too much. 


25. If you are NOT planning to take your math AP exam, please indicate whether the following 
statements are true or false. 


Name of math AP course: 


I don’t see any benefit in taking AP exams. 


I am not ready to take my math AP exam. 


I can afford to take the math exam. 


I am taking too many AP exams. 


My course load is too heavy to prepare for another test. 


The college I’m going to doesn’t accept AP scores. 


The cost of all of my AP exams is too much. 
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26. If you are NOT planning to take your English AP exam, please indicate whether the following 
statements are true or false. 


Name of English AP course: 


I don’t see any benefit in taking AP exams. 


I am not ready to take my English AP exam. 


I can afford to take the English exam. 


I am taking too many AP exams. 


My course load is too heavy to prepare for another test. 


The college I’m going to doesn’t accept AP scores. 


The cost of all of my AP exams is too much. 


Awards 


In this section, we want to know how students can earn rewards for successful enrollment and 
completion of AP math, science, and English courses for this school year. Please note that we are asking 
about all the rewards that have been offered, whether or not you have received them. 


27. What are the rewards offered to encourage you to participate in AP math, science, and English 
courses? (Choose all that apply.) 

C1 Scholarship incentives 

C1 Cash rewards 

L Prizes (e.g., classroom prizes, door prizes) 

L Weighted grades (e.g., 0.5 points additional for AP courses) 

CL) AP course exam fee is waived or supplemented 

CL) Other 

Please Specify: 

CL None (Skip to end of survey.) 
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28. How can students earn rewards? (Choose all that apply.) 

XO) Enrolling in an AP math, science, or English course 

CL) Completing and passing an AP math, science, or English course 

C1 Taking the AP exam 

CL) Passing the AP exam 

CL) Completing a certain number of AP-related tasks (such as online Study Sessions) 
Other 

LX Please Specify: 


29. How important are these rewards in encouraging you to participate in AP math, science, and English 
classes? (Choose one.) 

O Extremely important 

O Somewhat important 

O Slightly important 

O Not at all important 


30. Have you received your awards (if any) in a timely manner? (Choose one.) 
O Yes 
O No 


You are finished with the survey. Thank you for participating! 
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Appendix E: Teacher Interview Questions 


AP Teacher Interview 

To the interviewee: Thank you again for taking the time to talk with me today. This interview is 
expected to take about 30-45 minutes. We will focus on your thoughts on important components 
of the CRP, effects of CRP on school culture, and relative effectiveness of elements of the CRP 
on student success and interest in AP. 

If it is ok with you, I would like to record this interview. This recording would only be used to 
ensure that I capture your responses completely and accurately, and would not be accessible to 
anyone outside our research team. In addition, your name will not be linked with any response 
you give in any public reporting. However, based on your role, it might be possible to determine 
your identity, so we cannot guarantee complete anonymity in reporting. Do I have your 
permission to record this interview with you? 

[Note to interviewer: If the respondent does not want to be recorded, take notes, but do not 
proceed with recording. If the respondent agrees to be recorded, start the recording and repeat the 
question so that the positive response to this question and subsequent responses are recorded. ] 


a. What were the most effective or important parts of the CRP? 
b. How has school culture changed since the implementation of CRP? 


c. Are you satisfied with the level of training and support you have received 
throughout the school year? 


d. Did you attend the student study sessions? 
a. If yes, were they helpful to you? Please describe. 
e. Was mentoring offered to you as part of the CRP? 


a. If yes, did you take advantage of it? 
b. Was it beneficial to you? 


f. Do you feel adequately prepared to teach your AP course? 


g. What supports (e.g., increased planning time, additional PD, increased access to 
collaboration etc.) were offered to you specifically as an AP teacher? 


h. Were incentives offered to you as an AP teacher? 
a. If yes, please describe 


i. For what milestones or achievements were incentives offered? 
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How important were these incentives in encouraging your teaching of AP classes? 


Do you think that the monetary incentives offered to students have increased 
interest and enrollment in AP courses at your school? 


Would you change anything about the CRP to improve AP education in your 
school? 
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Appendix F: Administrator Interview Questions 


School Administrator Interview 

To the interviewee: Thank you again for taking the time to talk with me today. This interview is 
expected to take about 30-45 minutes. We will focus on your thoughts on important components of the 
CRP, effects of the CRP on school culture, and relative effectiveness of elements of the CRP on 
student success and interest in AP. 

If it is ok with you, I would like to record this interview. This recording would only be used to ensure 
that I capture your responses completely and accurately, and would not be accessible to anyone outside 
our research team. In addition, your name will not be linked with any response you give in any public 
reporting. However, based on your role, it might be possible to determine your identity, so we cannot 
guarantee complete anonymity in reporting. Do I have your permission to record this interview with 
you? 

[Note to interviewer: If the respondent does not want to be recorded, take notes, but do not proceed 
with recording. If the respondent agrees to be recorded, start the recording and repeat the question so 
that the positive response to this question and subsequent responses are recorded. ] 


a. What were the most effective or important parts of the CRP? 

b. How has school culture changed since the implementation of CRP? 

c. Did you attend a CRP-related goal setting meeting at your school? 
a. If yes, was the meeting an effective way to establish expectations for the 

program? 

b. What would you do differently? 

d. Did your school receive funds through the CRP for new equipment and materials? 
a. If yes, please describe how the materials impacted instruction 


e. Were incentives offered to you as the CRP Designated Administrator? 


a. If yes, please describe 
b. For what milestones or achievements were incentives offered? 


f. Do you think that the monetary incentives offered to students have increased interest 
and enrollment in AP courses at your school? 


g. Do you perform classroom observations? 


a. If yes, how does CRP classroom instruction and participation compare to non- 
CRP classes? 
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h. Would you change anything about the CRP to improve AP education in your school? 
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Appendix G: Student Survey Data 


Science 
Statement Biology Chemistry Env Science Physics overall 
TUE SOY Seeley 3.14 (0.72) 3.02, 0.87) 3.18 (0.69) 2.91 (0.82) 3.02 (0.79) 
confidence 
Improved my content understanding 3.11 (0.78) 3.00 (0.87) 3.26 (0.69) 2.98 (0.81) 3.05 (0.80) 
Conveniently scheduled 2.83 (0.85) 2.86 (0.97) 3.18 (0.72) 2.88 (0.92) 2.88 (0.89) 
inereased my conlidence in my ability 3.99 (0.80) 2.91 (0.93) 3.20(0.76) 2.85 (0.86) 2.97 (0.85) 
Improved my test-taking strategies 2.99 (0.89)  2.85(0.92)  3.10(0.74) 2.86(0.88) 2.93 (0.87) 
Computer 
Statement Calculus Science Statistics Math overall 
The study sessions increased my confidence 3.24 (0.80) 3.02 (0.64) 3.11 (0.74) 3.16 (0.77) 
Improved my content understanding 3.23 (0.68) 3.09 (0.76) 3.15 (0.68) 3.18 (0.69) 
Conveniently scheduled 3.07 (0.83) 2.63 (0.95) 2.96 (0.85) 2.97 (0.86) 
ee 3.18 (0.77) 2890.71) 3.03.81) 3.09 (0.78) 
Improved my test-taking strategies 3.14 (0.77) 2.91 (0.86) 3.01 (0.82) 3.06 (0.81) 
English English English 
Statement Language Literature overall 
The study sessions increased my confidence 3.15 (0.73) 3.13 (0.76) 3.14 (0.73) 
Improved my content understanding 3.18 (0.73) 3.13 (0.69) 3.16 (0.72) 
Conveniently scheduled 3.00 (0.89) 2.95 (0.81) 2.98 (0.87) 
Increased my confidence in my ability to take the AP exams 3.10 (0.77) 3.07 (0.71) 3.09 (0.76) 
Improved my test-taking strategies 3.17 (0.78) 3.07 (0.73) 3.14 (0.77) 
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Appendix H: Distribution of Teacher Survey Respondents by State, District, and School 


State District School n % Cumulativen Cumulative % 
CA school 1 3 2.33 3 2.33 
CA CA district 
CA school 2 5 3.88 8 6.2 
GA school 1 6 4.65 14 10.85 
GA GA district 
GA school 2 5 3.88 19 14.73 
IL school 1 2 1.55 21 16.28 
IL IL district IL school 2 3 2.33 24 18.6 
IL school 3 4 3.1 28 21.71 
LA LA district LA school 1 7 5.43 35 27.13 
MI school 1 9 6.98 44 34.11 
MI MI district 
MI school 2 7 5.43 51 39.53 
MO school 2 4 3.1 55 42.64 
MO MO district MO school | 4 3.1 59 45.74 
MO school 3 7 5.43 66 51.16 
ND district 1 ND school | 4 3.1 70 54.26 
ND ND school 2 8 6.2 78 60.47 
ND district 2 
ND school 3 5 3.88 83 64.34 
OH school 2 1 0.78 84 65.12 
OH school 1 2 1.55 86 66.67 
OH OH district 
OH school 3 3 2.33 89 68.99 
OH school 4 2 1.55 91 70.54 
PA district 1 PA school 1 4 3.1 95 73.64 
PA PA district 2 PA school 2 8 6.2 103 79.84 
PA district 3 PA school 3 7 5.43 110 85.27 
TX school 3 2. 1.55 112 86.82 
TX school 1 1 0.78 113 87.6 
TX TX district TX school 4 4 3.1 117 90.7 
TX school 2 6 4.65 123 95.35 
TX school 5 6 4.65 129 100 
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